In April of 2021 when I wrote my first blog post about execInstalledOnly there was zero information on what this setting did or how we were supposed to use it. Even VMware’s own Knowledge Base and Docs sites didn’t have any information on it. There was a Docs page around how to ‘enforce’ the execInstalledOnly setting, but this is (as you will soon learn) not the same thing as enabling the setting itself. The vSphere Security Configuration Guide 7.0 did recommend us to enable the setting, but this recommendation was later retracted when VMware discovered their own component for Update Manager Baselines was being blocked by execInstalledOnly.
If you need a basic lesson on what execInstalledOnly does, please click the link to my first blog post above and jump to the specific section to read up on it.
The Good:
Fast forward 19 months to November of 2022. ESXi 8.0 has been released in GA and it has execInstalledOnly enabled by default, which is mentioned in the Release Notes and Technical Blog. It has also been added back to the recommendations in the new vSphere Security Configuration Guide 8, which is great.
What does the execInstalledOnly setting do? If we try to execute a file that did not arrive in a VIB, execInstalledOnly will block it from being executed:
If someone turns off the default execInstalledOnly setting, this is displayed as a warning in the vCenter and ESXi web interfaces as well as logged to the hostd.log, vobd.log, and vmkernel.log:
ESXi will also log all attempts to execute non-VIB files, both the successful ones and the blocked ones:
So far so good, right? The above warnings give us lots of good information to work with, at least as long as we have central logging and a SOC that knows what to look for. I’ll summarize the strings to alert on at the end of this post.
The Bad:
The execInstalledOnly setting in ESXi 8.0 is implemented in two different ways:
- As a boot/kernel setting, in the same manner as it has been implemented in vSphere 6.x and 7.0. This one is still disabled by default, just like in the previous versions.
- As a runtime setting, which is new in vSphere 8.0. This is the one that is enabled by default, both for fresh installations and for upgrades.
To add to the confusion, there is also the enforcement of the execInstalledOnly setting. This is a new feature that was added in vSphere 7.0 U2 and is part of the ‘Managing a Secure ESXi configuration‘. It allows us to utilize UEFI Secure Boot and TPM 2.0 to add an extra check at boot-up which validates that the execInstalledOnly setting is still enabled, and intentionally purple screens the ESXi host if it is not.
This has led to a lot of confusion, including in VMware’s own Security Configuration Guide version 8, which still mentions the boot/kernel setting and claims it is enabled by default:
The Ugly:
So, what is wrong with this ‘runtime’ implementation? The primary problem is that it is way too easy to disable it. All an attacker needs to do is to add a single CLI command before executing their ransomware code and it will run just fine:
There is also another way to get around the execInstalledOnly protection, and that is to use the python executable that is packaged and installed in ESXi by VMware. This is already used by some ransomware gangs, who use python ransomware code instead of a regular executable ransomware.
Python can also be used in a sneaky way to put a persistent backdoor on ESXi hosts. This can fortunately fairly easily be prevented by enabling Secure Boot in the ESXi physical server, which stops any kind of added code to run during boot.
How do we protect ourselves against ransomware execution in ESXi 8.0?
The short answer is: If the attacker has root access to your ESXi hosts, there is not much we can do to stop them. All the protections that we put in place using root privileges can in some way be removed by anyone with root privileges. It is however important to understand that some protection is a lot better than no protection, and these features will give us some extra time and some very good detection capabilities.
Recommendation 1: One thing I do recommend which will slow the attacker down and increase your chances of detecting the attack is to use the good old execInstalledOnly boot/kernel setting rather than the ESXi 8.0 runtime setting. This will require the attacker to both disable the setting and reboot each ESXi host, which will hopefully buy you some time and set off some alarms in your SIEM/SOC (more info below).
Here is what happens if the attacker tries to disable the execInstalledOnly boot/kernel setting and not reboot the ESXi host before executing their ransomware:
After rebooting the ESXi host, the ransomware will execute normally, as long as the runtime execInstalledOnly setting and the enforcement of the execInstalledOnly setting have also been disabled.
Recommendation 2: Another effective and easy security measure to put in place is (as mentioned above) UEFI Secure Boot. It prevents different kinds of attempts at tampering with the ESXi boot files and gaining persistence across reboots. For more details on how to enable Secure boot, see my first blog post about it.
Recommendation 3: The only way (in my humble opinion) to have a chance at detecting and disarming the attackers before they succeed in escalating and executing the information stealing and/or ransomware encryption is to have 24/7 SOC monitoring of both your endpoints using EDR and your vSphere environment using syslog (and lots of filtering).
Bottom Line Recommendation: Since the above measures can partially be disabled and thus don’t provide 100 % protection against ransomware/malware execution in ESXi, we ultimately need to focus on the main goal: Don’t let attackers get root access to your ESXi hosts. This sounds fairly obvious, but given the number of methods the attacker can use to get there, it’s not quite that easy.
How do we prevent an attacker from getting root access to ESXi?
This will be described in more detail in future blog posts, but if we look at the most common attack paths, we need to make sure we meet the following goals:
- Don’t let the attacker get Administrator access in vCenter Server – If they do they can easily escalate it to ESXi root privileges on all hosts and encrypt the entire environment.
- Keep ESXi and vCenter Server updated with all important security patches – Unpatched security vulnerabilities are great shortcuts for taking control of ESXi or vCenter Server.
- Don’t use Active Directory users or groups for authenticating Administrator-level roles in vCenter Server – Don’t let an AD breach also become a vSphere breach.
- Segment your vCenter Servers and ESXi hosts – Make sure they can’t be reached from the regular servers and the client computers that are used for web and email.
- Manage your root and administrator passwords securely – Don’t use simple passwords, don’t reuse passwords and do keep them in a well-managed password vault or PAM system.
- Monitor your environments 24/7 – This is the only way to have a chance to detect and disarm an attack before it escalates and succeeds.
I hope this blog post has cleared some of the confusion around execInstalledOnly in vSphere 8.0 and that it can perhaps even inspire VMware to figure out a good way to protect the setting even better against tampering.
I do recommend you to read my other blog posts to get a broader view of how to architect and configure your vSphere environments to better withstand ransomware and other threats.
Appendix: Log messages related to execInstalledOnly on ESXi 8.0 that a SOC should keep their eyes on
These messages appear in different shapes in multiple files (hostd.log, vmkernel.log, vmkwarning.log, keypersist.log etc), but I’ll focus on what appears in vobd.log, since it summarizes the most important ones:
[vob.uw.exec.installonly.violation] Execution of non-installed file prevented:
[FILENAME] – File was not executed since execInstalledOnly was enabled. This is not necessarily malicious, but should be monitored and investigated. Non-VIB files should never be executed in ESXi, since it’s not a general-purpose operating system that should run third-party agents or utilities.
[vob.uw.exec.installonly.disabled] ExecInstalledOnly has been disabled. This allows the
execution of non-installed binaries on the host. Unknown content can cause
malware attacks similar to Ransomware. – The execInstalledOnly setting was disabled. This is definitely something that should render an alert and an investigation.
[vob.uw.exec.installonly.warning]
Execution of non-installed file: [FILENAME] – File was executed since execInstalledOnly was disabled. Unless you have intentionally disabled execInstalledOnly, this should be alerted and investigated.