The Exploit
In simple terms, the exploit takes advantage of a controlling mechanism in a function (in the software) responsible for evaluating the access to the REST API functionality of the affected devices. This functionality has hardcoded parameters used for authenticating the internal API requests made by the device.
By adding the “Forwarded” http header field (normally used by proxies to preserve/attach information about the original source of a request) to the request, and altering the “User-Agent” field, the controlling mechanism can be tricked to believe that the request originates from the device itself. Effectively, this leads to an authentication bypass giving an unauthorized attacker administrative privilege on the device.
The only parameters needed to use this are the following:
- A working REST API URL
- “User-Agent” field set to “Report Runner” or “Node.js“
- “Forwarded” field added and localhost IP address set as the “for” parameter
Below is an example of a working exploit:
GET /api/v2/cmdb/system/admin/admin/ Host: 172.16.1.20 User-Agent: Report Runner Forwarded: for=127.0.0.1 Accept: application/json, text/plain, */*
By exploiting this vulnerability, a threat actor can bypass authentication and execute a number of high privilege actions such as creating new local accounts, adding SSH key to the administrator accounts for persistence, configuring policies to gain remote network access to internal systems. In addition to this, the threat actor could mount other attacks against the rest of the IT-environment with the foothold and knowledge gain through the exploitation of this vulnerability.
Reference the Fortinet PSIRT advisory for more information about affected software and products.
First Things First
The findings in this post are based on the Truesec CSIRT experience from real life incidents with this vulnerability and tests using the POC presented by researcher James Horseman at Horizon3. The test environment is made up of a Fortigate virtual firewall running FortiOS v7.2.1 build1254.
Let’s start with some easy to digest bullet points:
- This vulnerability can be exploited even though the firewall administrator has not generated an API token
- The GUI uses API calls to communicate with the backend of the affected device, all those APIs (in theory) and most of the operations available in the REST API guide for the vulnerable products can be utilized in an attack
- Unnoticed configuration changes made by an attacker will persist a firewall upgrade
- The source IP address of an attacker will not be revealed when this exploit is utilized (see figure 2)
- It does not seem to be possible to change the password of the system administrator accounts, nor add new administrator accounts
- The logs associated with this exploit will have the user column set as “Local_Process_Access”
- Assume Breach if your system is vulnerable and sufficient logging is lacking (more info about logging can be found below)
Available Logs
When investigating a potential breach there are primarily two types of interesting system even logs: “General System Events” and the “REST API Events”.
These logs are found under the “Log & Report/System Events” menu in the firewall management GUI.
The “General System Event” logs contain information about configuration changes and the administrator account name executing the changes. These logs will not show information about any unauthorized GET requests (i.e., viewing part of configuration etc.).
The “REST API Events” logs contain information about all the API calls conducted by the system or the users. These logs show the URL paths, methods, and response code for the logged requests. The API event logs do however not show any information about the parameters (i.e., the body part in POST or PUT requests) sent in a request.
Ideally, both above-mentioned log types should be available during an investigation. If none of the event types are available in a vulnerable and exposed system, compromise should be assumed, and the situation should be handled as a cyber incident. In such a scenario, it will be impossible to assess whether the firewall configuration has been accessed by a threat actor or not, this is why a breach should be assumed.
It should also be noted that the “REST API Events” logs seem to be disabled by default, this was at least the case in our test environment. This means that many affected organizations will have to rely on events logged by the “General System Events” logs.
The Indicators of Compromise
When searching the web for information about this vulnerability, a log event indicating compromise is commonly shared amongst concerned IT-professionals. The log event message states: “System config file has been downloaded by user Local_Process_Access via Report Runner”.
This log message can be found in the “General System Events” if the system is compromised. In figure 2, log events generated by two separate malicious downloads are highlighted with red-colored boxes. This can be compared to the log events highlighted with green-colored boxes, which show that the firewall configuration has been downloaded by the authenticated administrator user “admin”.
The log messages in figure 2, indicating malicious configuration downloads, are generated by the following request:
GET /api/v2/monitor/system/config/backup?destination=file&file_format=fos&scope=global HTTP/2 Host: 172.16.1.20 User-Agent: Report Runner Forwarded: for=127.0.0.1 Accept: application/json, text/plain, */*
The response to a successful malicious request will contain the firewall configuration. It is not the detailed configuration output (i.e., the show full-configuration command) that is returned, but it will give a threat actor enough information about the environment to mount further and directed attacks.
The appearance of this log event in a vulnerable system indicates that it has been compromised and the IT-environment should therefore be thoroughly investigated. Ideally, the firewall should be brought offline while the investigation is conducted, and the suitable mitigation actions are executed. Mitigation actions may vary depending on the environment and the dependencies the compromised system has to vital, internal systems (i.e., Active Directory, Radius servers, SAML-connections and so on).
At a minimum, password resets (for admin accounts, local accounts, radius/VPN PSK etc.) should be forced and remote access features such as Client VPN should be protected with Multi Factor Authentication solutions.
Indications of Further Compromise
In a system with full logging enabled, it is possible to follow the steps of an attacker as (almost) all the events will be logged
To generate logs in our test environment, a new local user (user_3) was created, and later the password of the same user was changed, through the exploitable vulnerability in the firewall software. Figure 3 shows the “REST API Events” logs generated by these actions.
As seen, the logs only show the API URLs called, the method, and status code. Based on the values seen in the “Method” column, the logs indicate that changes were made but they do not show what these changes were.
Figure 4 shows the “General System Events” logs related to the same actions.
From these logs it is possible to understand that user_3 was added and that the user attributes were edited. It is however not possible to see what was edited. To further understand the changes made, the configuration (show full-configuration) and the raw logs need to be analyzed (looking for configuration attribute changes, correlating password time dates with log events etc.).
In environments where remote logging is not enabled, the logs will be saved locally on the compromised devices. At best, logs will be saved to disk, which might be limited in space, leading to important log entries being overwritten. In worst case scenarios, the compromised system might be configured to save logs in memory. Logs saved in the device memory will not survive a reboot. Unfortunately, in the upgrade process of solving the exploitable issue, systems that only have memory logging enabled might lose all the logs from a possible compromise. For this reason, in situations where logs are not available, affected organizations should make sure that logs are exported and secured prior to installing the software patch addressing the vulnerability. Otherwise, the forensics investigation will be much harder as it will have to rely solely on configuration analysis.
When a System Helps the Attacker to be Stealthy
Some of the configuration changes which can be made through this exploit might not render a log event and given the fact that “REST API Events” logs must be enabled manually, any unauthorized action resulting in a GET request will remain unseen (with the exception of the configuration download). This means that the information gathering phase of an attack may go by unnoticed as an attacker could gather the configuration through multiple, more specific API calls. This type of approach will not generate log events in the “General System Events“, this in contrast to the logs indicating a malicious download of the whole configuration.
As stated in the first Horaizon3 blog post, it is not possible to change the password of an existing administrator account, it is however possible to upload a ssh-key for an account, thus bypassing the need for a valid password. Adding a ssh-key to an account is de facto a change, but it will not be logged in the “General System Events”, which is a bit odd. To investigate possible ssh-key uploads, the configuration must be analyzed. The following firewall CLI command can be used for that:
show full-configuration | grep -f ssh-public-key
If a SSH key is present, it will be shown in the output of the command. See figure 5.
It is not possible to add an ssh-key to non-administrative accounts (i.e., the local accounts), therefor the analysis can exclude those accounts.
Another way to analyze possible configuration changes is to compare the current configuration with a backup configuration, this approach will however require an existing, up to date backup configuration file to compare with.
Configuration comparison should in such cases be focused on finding differences related to local accounts, firewall policies, or VPN-profiles. Basically, any part of the configuration enabling remote access to the internal network should be reviewed and analyzed.
Furthermore, if an attacker changes the password for an existing local account, the date of the password change will be recorded in the configuration. However, it is important to understand that the password change date can easily be manipulated and altered by an attacker. See figure 6
Help! We Are Compromised
In case your organization is compromised, or you suspect that your organization has been compromised, here’s what you should do:
- Take the compromised system offline, yes this will most probably result in a service outage, but it is necessary
- Secure the logs
- Make sure you Secure the logs in the previous step
- Upgrade the software
- Remove any configuration enabling access to the management from Internet
- Change all the passwords and pre-shared keys configured on the compromised device
- Remove all SSH keys configured for administrator accounts if their authenticity cannot be ensured
- Investigate the impact of the compromise; it is highly recommended to contact a professional investigator/IR-team as a threat actor may have moved laterally and expanded their network access to adjacent systems
- Make sure that your remote access solution enforces MFA
- Document all the integrations and the dependencies between the compromised device and other IT-systems and ensure mitigative actions are taken to secure them
- Consult a professional IR-team before going back to full production, otherwise there might be circumstances, and dependencies involved which could remain undetected
Conclusion
A vulnerable system exposed to the Internet has most probably been scanned and thus compromised. Compromised in this context means that the current configuration containing password hashes, account information, policies, network and addressing schemas are in the possession of a threat actor. If sufficient logs are not available, then the only sensible conclusion is to assume breach and handle the situation accordingly.
The investigation of a breach and the recovery steps needed to mitigate further attacks and minimize impact may vary depending on the environment and the integrations between the firewall and other vital IT-systems. This process must consider the fact password hashes in the leaked configuration can be subjected to offline cracking attacks with tools such as hashcat. Account names present in the configuration can be used in for instance password spraying attacks on other services exposed to the Internet, which can be very endangering if the same account names and passwords are used throughout the IT-environment. Furthermore, the leaked configuration might reveal information about internal naming conversions, which can facilitate further attacks
For instance, the recovery process is probably easy if the compromised firewall is only used for traffic filtering with a limited configuration set. The recovery process will however be much more complex and involve more parts of the IT-environment for a setup in which the firewall is used as Client VPN gateway with LDAP authentication of users, where the LDAP account configured is also a Domain Admin account.
It is also very important to upgrade all affected systems in your environment, even if their management interfaces are not exposed to the Internet. Vulnerable systems can be used for lateral movement or become part of an attack in other ways if a threat actor has gained foothold in your environment through some other entry point.
With the above-mentioned scenarios in mind, if you’re uncertain about the findings in your environment and how to interpret them, assume breach and contact professional incident responders to receive proper help with containment and recovery.