Background
Yesterday (September 29, 2022), Mandiant published some excellent research (Bad VIB(E)s Part One: Investigating Novel Malware Persistence Within ESXi Hypervisors) on a threat actor using a persistence method that utilizes VMware ESXi VIB (vSphere Installation Bundle) packages for “hiding” inside ESXi hosts and being able to relay commands and data to/from the threat actor. They have named them VIRTUALPITA and VIRTUALPIE.
This post describes how to protect your VMware vSphere environment against this threat, as well as future threats that utilize the same design flaw in ESXi to sneak in and hide. It also describes how to check for the threat in your vSphere environment.
How To Protect Your ESXi Hosts (the Short Version)
Enable UEFI Secure Boot for your physical ESXi hosts and make sure the VIB Acceptance Level setting hasn’t been lowered from the default ‘PartnerSupported’ to ‘CommunitySupported.’
These steps will prevent a threat actor from circumventing your set VIB Acceptance Level by simply typing –force when installing the non-signed untrusted VIB. Instructions on how to enable Secure Boot on your ESXi hosts can be found here.
Enabling Secure Boot includes running the pre-check (secureBoot.py) to make sure there are not any unsigned VIBs that will prevent it. Rather than running it manually on one ESXi host at a time over SSH, use VMwares provided PowerCLI script that will check multiple ESXi hosts in one go. The script can be downloaded from VMware KB #89619.
How To Protect Your ESXi Hosts (the Long Version)
ESXi is an appliance, so there’s very little reason to run any third-party binaries in it. There are many security features in ESXi that prevent foreign code from executing, but unfortunately, some are not enabled by default. That’s why I wrote my first blog post on the topic back in April 2021. It describes how you make use of TPM 2.0, UEFI Secure Boot, and execInstalledOnly, which makes ESXi only allow execution of binaries that originate from approved and signed VIBs. Link to that post: Secure Your VMware ESXi Hosts Against Ransomware
In theory, the default settings should be sufficient since the default VIB installation Acceptance Level is ‘PartnerSupported,’ which means that no unsigned or unapproved VIBs are allowed to be installed. Unfortunately, Mandiant’s write-up points to (in my opinion) a serious design flaw that allows a malicious user to override these security settings by simply adding the switch –force at the end of the VIB installation command.
Fortunately, by utilizing UEFI Secure Boot, this flaw can be mitigated and the –force switch disabled. This is verified by VMware in their guidance document for this threat: VMware KB #89619.
Other methods that should work for preventing this malware from gaining persistence and/or executing:
- The execInstalledOnly setting should stop the malware from executing, but we haven’t been able to verify this with the malware in question.
- Keeping ESXi hosts updated using vSphere Lifecycle Manager (vLCM) Images (as opposed to the old Update Manager baselines) should have spotted the extra VIBs since it’s declarative and will warn if the stipulated image doesn’t match the VIBs currently installed on the target hosts.
Note that to exploit this flaw, the threat actor first needs to get access to a root-level admin account on the ESXi host. The flaw doesn’t give the threat actor a way in, as a remote code execution (RCE) vulnerability would. In most cases, an attacker would use a newly gained root access to deploy ransomware and/or leak sensitive data from the customer rather than lay low and sneak in a hidden persistent backdoor to be able to act later in the future. This threat is therefore targeted at a very specific type of victim and used by a very specific type of threat actor, as indicated by Mandiant in the report.
How To Check Your Environment for This Threat
There are multiple ways to detect the presence of this persistent malware:
- Retroactively in the VIB list – This is performed using VMware’s PowerCLI script, which can be downloaded from VMware KB #89619. This is the recommended primary method to use, according to VMware.
- By scanning the ESXi file system and/or memory for the malware. This method is described by Mandiant in their second blog post: Bad VIB(E)s Part Two: Detection and Hardening within ESXi Hypervisors
- At installation time in the ESXi logs – This method is described below for reference. Note that this requires intact logs from the time of installation, which might be far back in time. Local logs are always susceptible to log manipulation from the malware since it is persistent on the ESXi host.
/var/log/vobd.log will contain log messages similar to the following, indicating the ‘Acceptance level’ has been overridden:
2022-09-30T08:16:35.540Z: [UserLevelCorrelator] 4595327941us: [esx.audit.esximage.install.novalidation] Attempting to install an image profile with validation disabled. This may result in an image with unsatisfied dependencies, file or package conflicts, and potential security violations. 2022-09-30T08:16:35.540Z: [GenericCorrelator] 4595327702us: [vob.user.esximage.install.novalidation] Attempting to install an image profile with validation disabled.
/var/log/hostd.log will also contain similar log messages (hostname redacted):
2022-09-30T08:16:35.545Z info hostd[68444] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 145 : Issue detected on [hostname] in ha-datacenter: Attempting to install an image profile with validation disabled. This may result in an image with unsatisfied dependencies, (2022-09-30T08:16:35.544Z cpu0:70678) 2022-09-30T08:16:35.545Z info hostd[68444] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 146 : Issue detected on [hostname] in ha-datacenter: Attempting to install an image profile bypassing signing and acceptance level verification. This may pose a large security r (2022-09-30T08:16:35.544Z cpu0:70678) 2022-09-30T08:16:35.658Z info hostd[67771] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 147 : SECURITY ALERT: Installing image profile '(Updated) ESXi-7.0U3c-19193900-standard' with acceptance level checking disabled.
/var/log/esxupdate.log will have the following messages (hostname and VIB name redacted):
2022-09-30T08:16:34Z esxupdate: 69525: root: INFO: Options = {'depot': None, 'viburl': ['/vmfs/volumes/datastore1/VIB-test/[VIBNAME].vib'], 'nameid': None, 'profile': None, 'baseimageversion': None, 'addon': None, 'softwarespec': None, 'level': None, 'updateonly': False, 'noliveinstall': False, 'nomaintmode': False, 'force': True, 'dryrun': False, 'oktoremove': False, 'proxy': None, 'nosigcheck': False, 'pending': None, 'rebooting': False, 'downgrade': None, 'nohwwarning': False}
2022-09-30T08:16:35Z esxupdate: 69525: HostImage: WARNING: SECURITY ALERT: Installing image profile '(Updated) ESXi-7.0U3c-19193900-standard' with acceptance level checking disabled.
2022-09-30T08:16:35Z esxupdate: 69525: vmware.runcommand: INFO: runcommand called with: args = '['/usr/lib/vmware/vob/bin/addvob', 'vob.user.esximage.install.securityalert', '(Updated) ESXi-7.0U3c-19193900-standard', 'acceptance level checking disabled']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2022-09-30T08:16:35Z esxupdate: 69525: HostImage: WARNING: VIB [VIBNAME] checksum is not available, skip verification