Secure Your VMware ESXi Hosts Against Ransomware

Why we should use execInstalledOnly to protect ESXi against ransomware

• Ransomware executing inside a VMware vSphere ESXi host can encrypt all the virtual machines at once, without having to compromise each guest operating system. An example of this is RansomEXX a.k.a. Defray777. More info on it can be found in this Crowdstrike writeup.
• This attack vector is possible because once attackers get control of an ESXi host, they are by default allowed to upload and execute any custom binaries they want.
• We can fairly easily prevent this by using the relatively unknown ESXi setting VMkernel.Boot.execInstalledOnly (optionally in combination with TPM 2.0 and UEFI Secure Boot) which is described in the ‘Three steps to protect ESXi against ransomware’ section below. (Jump straight there)

Background

We have recently seen an increase in ransomware attacks where the encryption is executed from the virtualization platform (ESXi or Hyper-V hosts) rather than from inside each guest operating systems (Windows, Linux etc). The benefit of this method from the attackers’ side is that they can encrypt numerous systems without having to reach them all over the network and obtain administrative privileges. This can greatly increase the scope and speed of the attack, which is bad news for us.

This blog post won’t go into the technical details on how the attacker gets into the ESXi hosts to execute the actual ransomware. This could for example be done through an RCE vulnerability such as the one for SLP in ESXi or through Active Directory->vCenter Server->ESXi, but also in other ways. A future blog post will analyze this in more detail and provide more suggested protections.

The ransomware will encrypt all virtual machines’ vmdk files on all attached datastores. It will also encrypt the ESXi host itself including all log files, so unless you have central tamper-proof logging in place it will be very difficult to secure forensic evidence regarding how the attack was carried out.

Despite the encryption, the ESXi hosts will usually remain running since they have already loaded the system files into memory. However, they will usually not survive a reboot, and will need a complete reinstallation. As long as the host is still running, the ransomware monitors the virtual machines and will encrypt any new vmdk or other virtual machine files that are put on shared datastores that it can reach.

VMware has a good technical post about this ransomware at Deconstructing Defray777 Ransomware, which goes through the technical details, but doesn’t mention specifically how to protect the ESXi hosts.

They also have some good videos covering the basics of ransomware protection on vSphere (but doesn’t mention execInstalledOnly):

When we at Truesec perform Security Health Checks of customers’ vSphere environments, we always give everyone the following fundamental recommendations, so do make sure you also work towards getting these under control:

• Keep your systems (vCenter Server, ESXi hosts, VMware Tools etc.) up to date when there are security patches released.
• Use unique, strong passwords for administrative accounts and handle them securely. Consider not using Active Directory for administrator level access to vSphere.
• Segment your networks so that vCenter Server and ESXi administrative interfaces are not reachable for non-administrative computers and users. Use dedicated workstations and MFA for administrators.
• Configure central logging so that you have tamper-proof logs of all administrative actions and changes in your environment.
• Make frequent backups, and make sure they can not be deleted even if the attacker gets complete control over the rest of your environment.

Three steps to protect ESXi against ransomware

To prevent the actual ransomware execution, we recommend our customers to take the following steps for all ESXi hosts:

• (Optional) Use TPM 2.0 chips
• Enable UEFI Secure Boot on the physical servers (I used to have this one as ‘optional’, but now it’s highly recommended due to flaws in settting an ‘Acceptance level’ alone)
• Prohibit execution of custom code inside ESXi (VMkernel.Boot.execInstalledOnly)

As always, make sure you test and evaluate the consequence of any upgrades and changes on a non-critical part of your environment before rolling it out in production. Also make sure your third-party vendors support the new configuration.

(Optional) Step 1: TPM 2.0

TPM 2.0 is a hardware chip that most modern physical servers have. It allows the operating system (ESXi in our case) to store secrets, keys, measurements etc. in a secure manner. This is used by vCenter Server to make sure the ESXi hosts’ boot files haven’t been tampered with, and works with vSphere/ESXi 6.7 and newer. More info here: https://blogs.vmware.com/vsphere/2018/04/vsphere-6-7-esxi-tpm-2-0.html

For the vSphere Attestation, there isn’t any specific configuration that needs to be set. If your ESXi hosts have active TPM 2.0 chips, vCenter Server will automatically display their attestation status in the ‘Monitor->Security’ tab of the clusters:

In vSphere 7.0 U2 and newer, the TPM 2.0 chip is also used to encrypt the configuration of the ESXi host as well as protect some settings from tampering (called ‘enforcement’). This is described in detail in the vSphere documentation.

Step 2: Secure Boot

Secure Boot is a UEFI BIOS feature that strengthens the security of the operating system (ESXi in this case) by making sure that all code that is loaded at boot is digitally signed and has not been tampered with. It also adds the benefit of disallowing threat actors from circumventing your set VIB ‘Acceptance level’ by them simply adding ‘–force’ at the end of the installation command. This was used to gain persistence in the case of VIRTUALPITA and VIRTUALPIE, see the separate blog post about that.

Unlike some other operating systems, ESXi can have Secure Boot enabled retroactively without having to perform a complete reinstallation.

However, there might be some installation packages (‘VIBs’, in vSphere language) that are not approved/signed by VMware or partners. These will then have the wrong ‘Acceptance level’ and can prevent Secure Boot from working correctly. To check if your ESXi host already has Secure Boot enabled, and whether there are any obstacles to enabling it, run the following two commands from an ESXi command line (SSH or ESXi Shell):

/usr/lib/vmware/secureboot/bin/secureBoot.py -s

/usr/lib/vmware/secureboot/bin/secureBoot.py -c

As we can see in the output above, Secure Boot is currently Disabled, but there are no obstacles preventing us from enabling it. We can now put this ESXi host in Maintenance Mode, reboot it, enter the server’s BIOS setup, enable Secure Boot, and boot up the ESXi host again. If you’re unsure on how to enable Secure Boot, check with your hardware server vendor on how to do this.

[UPDATE 2022-10-03]: To check multiple ESXi hosts for unsigned VIBs you can utilize VMware’s PowerCLI script released for detecting VIRTUALPITA and VIRTUALPIE. The script can be downloaded from VMware KB #89619.

In vSphere 7.0 U2, the Secure Boot setting can be protected from tampering using the ‘enforcement’ capability. This is set using the following command line:

esxcli system settings encryption set --require-secure-boot=TRUE

More information is available in this VMware Documentation page.

Step 3: execInstalledOnly

—
[UPDATE 2022-04-28]: IMPORTANT: If you are updating/patching ESXi hosts using vSphere Lifecycle Manager (formerly known as Update Manager) using the old fashioned Baseline method rather than the newer Image method (link to article describing the difference) you will bump into problems when having execInstalledOnly set to TRUE.

The recommended workaround is to switch to the Image method, since it will also bring other benefits. If you can’t switch, you will unfortunately need to wait until vSphere 8.0 before being able to enable execInstalledOnly.

When using the baseline method and enabling execInstalledOnly, the error message you will get when scanning an ESXi host for patch compliance is:

Cannot deploy host upgrade agent. Ensure that vSphere Lifecycle Manager is officially signed. Check the network connectivity and logs of host agent and vpxa for details.

If you are getting this error message, either switch to the vLCM Image method or follow the instructions in https://kb.vmware.com/s/articl… to revert the execInstalledOnly setting and the enforcement of the setting.

If you set execInstalledOnly back to FALSE but keep the enforcement at TRUE, you will get a purple screen when rebooting the ESXi host. The purple screen is by design, and is described at the end of this blog post.

# Revert the enforcement of the setting
esxcli system settings encryption set --require-exec-installed-only=FALSE 

# Revert the setting itself
esxcli system settings kernel set -s execInstalledOnly -v FALSE 

The reason for the Lifecycle Manager problem is that when using Baselines, VMware is apparently using an unsigned VIB, which is the update agent that Lifecycle Manager pushes out to the ESXi hosts when scanning or updating them. This is a very unfortunate mistake, which I hope they will fix before vSphere 8.0. I recommend you open a support case and tell VMware they need to fix this in 7.0 as well.
—
(Now back to the blog text..)

The execInstalledOnly setting prohibits execution of custom code inside ESXi and will make the ESXi host simply refuse to execute anything that was not installed through a signed VIB package from a certified partner.

This is very easy to achieve in ESXi compared to a general purpose operating system like Windows or Linux. ESXi is by design an “appliance” which doesn’t require any custom code to be run on it apart from VMware’s own code and the drivers and utilities of certified partners. Combining execInstalledOnly with TPM and Secure Boot which tamper-proofs the existing VIBs gives us an excellent combination of protective measures against ransomware and other malware executing inside the ESXi hosts.

The setting is found in ESXi under Manage->Advanced Settings at VMkernel.Boot.execInstalledOnly and it can be set without having to open a CLI to each ESXi host. We can set it for individual ESXi hosts using the vSphere Web Client or for multiple ESXi hosts at a time using PowerCLI (VMware’s PowerShell modules). The examples below use the CLI over SSH, since this gives us some additional information that is good to have when explaining the inner workings of the setting.

The default setting of an ESXi host is that execInstalledOnly is set to FALSE. We can verify this by simply listing the setting using

esxcli system settings kernel list -o execinstalledonly

and then checking that we are allowed to execute a custom binary. Rather than executing a real ransomware, we used a test binary that displays a ‘Hello world’ message to indicate that it was allowed to run:

Now, we set the execInstalledOnly setting to TRUE using

esxcli system settings kernel set -s execinstalledonly -v TRUE

Note that the ‘Configured’ value has changed to TRUE, but that the ‘Runtime’ value is still at FALSE. This is because the system requires a reboot for the setting to start working. We can verify that the protection is still not in effect by successfully executing our test binary again.

After rebooting the system, we can now see that the setting’s ‘Runtime’ value is TRUE, and we can also see that we are no longer allowed to execute our custom test binary.

Make sure you test your ESXi hosts properly after changing this setting on a test host. Some hardware vendors might have agents and/or utilities running inside ESXi that perform monitoring, central configuration, firmware upgrades etc.

In vSphere 7.0 U2, the execInstalledOnly setting can be protected from tampering using the ‘enforcement’ capability. This is set using the following command line:

esxcli system settings encryption set –require-exec-installed-only=TRUE

More information is available in this VMware Documentation page.

Putting it all together

Putting TPM, Secure Boot and execInstalledOnly together, we can now establish the following order of checks and changes to each ESXi host:
(Note that steps 1-3 and 6-7 are optional)

1. /usr/lib/vmware/secureboot/bin/secureBoot.py -s
2. /usr/lib/vmware/secureboot/bin/secureBoot.py -c
3. Make sure there are no warnings or errors above
4. esxcli system settings kernel set -s execinstalledonly -v TRUE (or set on multiple hosts using PowerCLI)
5. Put the ESXi host in Maintenance Mode and reboot it
6. At boot-up, enter the BIOS setup and enable UEFI Secure Boot (if not already enabled according to step 1)
7. (If running vSphere 7.0 U2 or newer and having a TPM 2.0 chip):
   1. esxcli system settings encryption set --require-secure-boot=TRUE
   2. esxcli system settings encryption set --require-exec-installed-only=TRUE
   3. /sbin/auto-backup.sh
8. Check that everything works, and take the ESXi host out of Maintenance Mode

Again, make sure you properly test these changes on non-critical ESXi hosts before rolling them out to the entire environment. Especially, test an ESXi update/upgrade using Lifecycle Manager to make sure it works well.

Once set, monitor any changes to these settings using central logging and set alerts so that you get notified if anyone tries to change them. Also monitor events around VMs getting unexpectedly shut down, since a ransomware needs to perform this step before being able to encrypt the files, since they’re locked by the hypervisor while running.

Tamper-proofing ESXi

If you’re using the vSphere 7.0 U2 additional protections to ‘enforce’ the execInstalledOnly setting and Secure Boot (step 7 above), your ESXi hosts will fail to boot and display a purple screen (which is good) if TPM, Secure Boot or execInstalledOnly is tampered with.

This is described in the first example in following VMware KB article: Boot time failures due to ESXi configuration encryption (81446)

Fix the problem by going back and re-checking all the above steps:

• If TPM 2.0 has been disabled, re-enable it.
• If UEFI secure boot has been disabled, enable it back.
• If execInstalledOnly boot option is set to FALSE, change it back to its initial value (i.e. TRUE).

If you have any questions or bump into any problems, don’t hesitate to contact us and we will try to help you out. Watch this space for more blog posts and videos about ransomware and vSphere/ESXi security.