TPM Encryption Recovery Key Backup Alarm - ESXi 7u2

Recovery Key Alarm 

This is A new alarm which triggered to prompt the administrator to back up the 
recovery key.

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-6B13C88E-B57A-46B7-B3EC-0EE3F6C4D346.html 

Login to the ESXi and note down the key.
 

 [root@host1] esxcli system settings encyption recovery list

Recovery ID                             Key
--------------------------------------  ---
{2DDD5424-7F3F-406A-8DA8-D62630F6C8BC}  478269-039194-473926-430939-686855-231401-642208-184477-602511
-225586-551660-586542-338394-092578-687140-267425

This is really important as this key is needed to recover the Host in case, 

• You cleared the TPM
• The TPM failed.
  

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-23FFB8BB-BD8B-46F1-BB59-D716418E889A.html

if you have the key recorded you can use below steps. 

Procedure

1. (Optional) If the TPM failed, move the disk (having the boot bank) to another host with a TPM.
2. Start the ESXi host.
3. When the ESXi installer window appears, press Shift+O to edit boot options.
   
4. At the command prompt, enter the boot option to recover the configuration.

   encryptionRecoveryKey=recovery_key

EVC - EVCAdmissionFailed - vCenter is running older version than the ESXi

This was a good case and I enjoyed finding the root case.

As per the logs, I can see that the ESXI servers were patched last year and some vibs were upgraded including CPU microcode which introduces new features based on the version.

host004.abc.com

 2020-04-21T04:20 The following VIBs are installed

 cpu-microcode          6.5.0-2.57.9298722

 

host2.abc.com

 2020-06-22T12:09:39.852806+00:00 The following VIBs are installed

 cpu-microcode          6.5.0-2.57.9298722

I was able to see that the VM TEST3. was rebooted after the upgrade, not any of the other TEST VMs.

5217ee1b9b11c87d-eef0a5e01b10badb/1a7df55b-d813-5f4a-2742-0cc47a660d00/vmware-32.log:2020-08-29T05:55 - Halt initiated from inside the Guest OS

5217ee1b9b11c87d-eef0a5e01b10badb/1a7df55b-d813-5f4a-2742-0cc47a660d00/vmware-33.log:2020-08-29T17 - Reboot initiated from inside the Guest OS

so when a VM restart it will boot with the available CPU capabilities in this case with the below additional capabilities for VM TEST3. 

cpuid.STIBP - Bool:Min:1

cpuid.FCMD - Bool:Min:1

cpuid.SSBD - Bool:Min:1

cpuid.IBRS - Bool:Min:1

cpuid.IBPB - Bool:Min:1

In the vmware.log of the VMs like TEST2 we can see that the requested capabilities were less than what available from the host wich was came from the OLD Host.

When we enabling the EVC to it will limit the capability of the CPU define by the EVC version and this is apply when a VM boots up.

Finally we come to the issue, the vCenter version (vCenter Server 6.5 Update 1e)  running was older than the ESXi version.

the issue is what vcenter know as the capabilities of Broadwell was less than what the Hosts can capable of(what TEST3 is running with). 

2021-03-12T11:23:37.192Z info vpxd[7F743BE7E700] [Originator@6876 sub=MoHost opID=EvcConfigFormMediator-validate-7760456-ngc:70214055-74] [HostMo::TestEvcModeInt] intel-broadwell EVC mode test on host2.wdc04c.

octousa.com failed: EVCAdmissionFailed: [N3Vim5Fault26EVCAdmissionFailedVmActiveE:0x00007f7534126fa0]

upgrading the vCenter to 6.5U2h resolved the issue.