This was a good case and I enjoyed finding the root case.
As per the logs, I can see that the ESXI servers were patched last year and some vibs were upgraded including CPU microcode which introduces new features based on the version.
host004.abc.com
2020-04-21T04:20 The following VIBs are installed
cpu-microcode 6.5.0-2.57.9298722
host2.abc.com
2020-06-22T12:09:39.852806+00:00 The following VIBs are installed
cpu-microcode 6.5.0-2.57.9298722
I was able to see that the VM TEST3. was rebooted after the upgrade, not any of the other TEST VMs.
5217ee1b9b11c87d-eef0a5e01b10badb/1a7df55b-d813-5f4a-2742-0cc47a660d00/vmware-32.log:2020-08-29T05:55 - Halt initiated from inside the Guest OS
5217ee1b9b11c87d-eef0a5e01b10badb/1a7df55b-d813-5f4a-2742-0cc47a660d00/vmware-33.log:2020-08-29T17 - Reboot initiated from inside the Guest OS
so when a VM restart it will boot with the available CPU capabilities in this case with the below additional capabilities for VM TEST3.
cpuid.STIBP - Bool:Min:1
cpuid.FCMD - Bool:Min:1
cpuid.SSBD - Bool:Min:1
cpuid.IBRS - Bool:Min:1
cpuid.IBPB - Bool:Min:1
In the vmware.log of the VMs like TEST2 we can see that the requested capabilities were less than what available from the host wich was came from the OLD Host.
When we enabling the EVC to it will limit the capability of the CPU define by the EVC version and this is apply when a VM boots up.
Finally we come to the issue, the vCenter version (vCenter Server 6.5 Update 1e) running was older than the ESXi version.
the issue is what vcenter know as the capabilities of Broadwell was less than what the Hosts can capable of(what TEST3 is running with).
2021-03-12T11:23:37.192Z info vpxd[7F743BE7E700] [Originator@6876 sub=MoHost opID=EvcConfigFormMediator-validate-7760456-ngc:70214055-74] [HostMo::TestEvcModeInt] intel-broadwell EVC mode test on host2.wdc04c.
octousa.com failed: EVCAdmissionFailed: [N3Vim5Fault26EVCAdmissionFailedVmActiveE:0x00007f7534126fa0]
upgrading the vCenter to 6.5U2h resolved the issue.
