How To Prevent Ransomware Attackers From Taking Over Your ESXi 8.0 Hosts

[UPDATE 2023-03-20]: Apparently this security feature wasn’t really meant to be a security feature. I discovered in my lab that this setting does prevent some kinds of password changes, but doesn’t prevent password changes using for example Host Profiles. I reported this to VMware Security Response Center a couple of weeks ago, and now they have investigated this with engineering and changed the text on the Docs page to highlight what it does and doesn’t do:

“When you deactivate the shell access for a user with administrative access, by virtue of being denied shell access, that user cannot grant shell access to other users, or change the passwords of users that have shell access. Other permissions, such as host profiles, will still allow users such as vpxuser and dcui to change passwords of other users.”

In summary: This security feature is great, but just like everything else it is not 100 % waterproof against attacks. I still recommend you implement it in your environments, since it will at least buy you some time, hopefully allow you to detect the attack earlier in your SIEM/XDR logs and make it more difficult for an attacker to get through the entire attack chain depicted below. I also recommend that you read and implement the other security features that I have blogged about.

How Is a Ransomware Attack Against vSphere/ESXi Usually Executed?

The attack chain (or “cyber kill chain,” as it is also referred to) of a ransomware attack against vSphere most often looks something like this:

1. An attacker gets a foothold on a server or a client in the environment. This can be achieved through, for example, unpatched vulnerabilities, configuration mistakes, or by phishing a regular user or administrator.
2. They manage to get the credentials of a Domain Admin account in Active Directory (AD). (Learn more about how to protect against this using AD Tiering in this video.)
3. If vCenter Server uses AD groups/accounts to authorize Administrator-level access, the attacker can now get access to vCenter Server with the “Administrator” role access to all objects, including the ESXi hosts managed by vCenter Server.
4. Using this role, the attacker can set a new root password on the ESXi hosts, disable Lockdown Mode if necessary, and start the SSH service on them.
5. The attacker connects to the ESXi hosts using SSH, uploads their ransomware code, and executes it. The ransomware stops all running VMs and encrypts all VM files in the Datastores. Game over. Call your CSIRT team.

Protecting the ESXi Root Passwords

Let’s zoom in on the last steps and work our way backward from the end: In a ransomware attack, the end goal of the threat actor (attacker) is to be able to log in to your ESXi hosts, upload the ransomware and execute it to encrypt all virtual machine files in the Datastores (Step 3 in the diagram below).

In order to log in and execute the ransomware, the attacker needs a username and password for a high-privilege user on each ESXi host. For the sake of simplicity, we’ll assume they’ll use the “root” account. They will also need the SSH service to be started and TCP port 22 on the ESXi hosts to be reachable from where they are.

Even if they don’t know the ESXi hosts’ root passwords, it has always been possible in previous vSphere versions for a user in vCenter Server that has the Administrator role on an ESXi host to use their privilege to simply set the ESXi root password to whatever they want, disable Lockdown Mode if enabled, start SSH and login to the ESXi host with the new password (Step 2 in the diagram above). This is the lateral step we can now prevent the attacker from taking.

Below is an example script that logs into vCenter Server, picks an ESXi host, sets a new password for root, disables Lockdown Mode, and starts the SSH service:

Connect-VIserver vcenter01.anders.local

$vmhost = Get-VMhost esxi01.anders.local
$username = 'root'
$newpw = 'P@ssw0rd01!'

$esxcli = Get-EsxCli -VMHost $vmhost -v2
$args1 = $esxcli.system.account.set.CreateArgs()
$args1.id = $username
$args1.password = $newpw
$args1.passwordconfirmation = $newpw
$esxcli.system.account.set.Invoke($args1)

($vmhost | get-view).ExitLockdownMode()

$vmhost | get-vmhostservice | where-object {$_.key -eq "TSM-SSH"} | start-vmhostservice -confirm:$false

Under the hood, moving from vCenter Server to ESXi is performed using vCenter’s service account in ESXi, named “vpxuser.” This account exists in all ESXi hosts by default and gets a password set by vCenter when an ESXi host is connected. This password is, by default, rotated every 30 days by vCenter.

Since vpxuser has the same role as root (“Admin” = Full access rights) as well as the “Shell access” permission, it can do pretty much everything the root account can do, including managing the configuration of ESXi, changing other accounts’ passwords, logging in over SSH, executing ransomware, etc.

This means that if the attacker has breached the vCenter Server with a root or Administrator-level user, there has previously not been any way to protect against them also breaching the ESXi hosts. So even if we have set good, strong, unique passwords on the ESXi hosts and managed them properly, it will have been in vain if the attacker manages to breach the vCenter Server. The best way to prevent the attacker from logging in has been by preventing them from reaching the ESXi hosts on TCP port 22 using a firewall of some kind.

Wouldn’t it be great if there was a way to let vpxuser manage and configure ESXi but prevent it from being able to manipulate the other local ESXi accounts (root + dcui) and also prevent it from being used to log in using SSH or ESXi shell?

ESXi 8.0 To the Rescue!

The good news is that ESXi 8.0 has introduced a way to prevent a vCenter Administrator-role user from changing the passwords on the ESXi hosts that vCenter Server manages. This is done by removing some of the permissions that the ESXi local vpxuser account has, specifically its permission called “Shell access.”

I initially spotted this in a VMware Docs page for vSphere 8.0 security called Assigning Privileges for ESXi Hosts. It had a new section compared to the 7.0 version called “Deactivating Shell Access for non-root ESXi Users.” This got me very interested, but unfortunately, it didn’t mention how to actually do it. After sending feedback on the Docs page (yes, they actually read the feedback) as well as asking around on VMware Communities and the vExpert Slack forum, Willam Lam came to the rescue, dug up the information internally, and wrote a great blog post about it. After that, the Docs page was also updated with the missing information.

If we list the local accounts in ESXi 8.0 using “esxcli system account list,” we will notice there is a new column called “Shell access”:

The way to remove the “Shell access” permission from vpxuser is by running the command “esxcli system account set -i vpxuser -s false.” This can also be performed for the dcui account:

Now if we attempt to change the ESXi root password using our vCenter Administrator role privileges, it will simply fail:

PS> $esxcli.system.account.set.Invoke($args1)

Message: EsxCLI.CLIFault.summary;
InnerText: Access to perform the operation was denied. Only users with shell access can make these changes.EsxCLI.CLIFault.summary
At line:1 char:1
+ $esxcli.system.account.set.Invoke($args1)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : OperationStopped: (:) [], MethodFault
 + FullyQualifiedErrorId : VMware.VimAutomation.Sdk.Types.V1.ErrorHandling.VimException.MethodFault

Since we have also removed the possibility for vpxuser to log in to the shell, the attacker can no longer fetch its encrypted password from the vCenter Postgres DB, decrypt it and use it for logging in to SSH or ESXi Shell directly either.

In summary, this is very good news for us and very bad news for the ransomware gangs. Even if they have completely breached Active Directory and/or vCenter Server, they will now have to find another way of getting into the ESXi hosts, for example:

• Brute-forcing the ESXi root password, or trying the same passwords that the administrators have used in AD and vCenter Server.
• Exploiting unpatched security vulnerabilities in ESXi.
• Hoping that the ESXi hosts are AD joined.
• Other methods, which I will not go into detail about.

Drawbacks?

So, are there any drawbacks or pitfalls when implementing this protection? Here are the ones I can think of initially:

• It will be a bit more cumbersome to manage and rotate the root passwords on multiple ESXi hosts.
• There will be a greater negative impact if we forget the root password since we won’t be able to reset it using our vCenter Server privileges.

  (Update 2023-03-20: Both of these will still be possible using Host Profiles and hopefully also the new vLCM Configuration Profiles feature that will GA in ESXi 8.0 Update 1)

Let me know if you experience any other drawbacks, and whether you still think it’s worth it compared to allowing ransomware attackers to encrypt all your virtual machines. 🙂

As always, test this setting in a lab environment and/or on a small number of ESXi hosts before rolling it out broadly!