“Running hypervisors inside hypervisors is the ultimate sandbox for testing cloud migrations. But if you forget to expose hardware VT-x/EPT CPU flags and block Promiscuous Mode on your physical vDS, your nested VMs will fail to power on or communicate.”

In June 2018, during my tenure as a VMware NSX Consultant at IBM Cloud, we were building an automated multi-tenant lab platform called the SDDC Pod Factory.

The goal was self-service staging: allow enterprise architects to click a button and spin up a complete, isolated multi-node SDDC sandbox—nested ESXi hypervisors, vCenter, vSAN, and NSX overlay networking—running inside virtual machines on physical bare-metal ESXi hosts.

It was the perfect environment for testing high-risk NSX-V to NSX-T migrations.

Until we tried powering on our first nested workload VM.

---

The Double Virtualization Barrier

Nested virtualization means running a hypervisor (Guest ESXi) inside a virtual machine managed by a physical hypervisor (Host ESXi).

When the nested Guest ESXi boots, it behaves like a physical server.

However, when Guest ESXi attempts to spin up a 64-bit virtual machine inside its own nested cluster, it requires direct access to the physical CPU’s hardware virtualization extensions: Intel VT-x (or AMD-V) and Extended Page Tables (EPT).

By default, VMware vSphere masks physical CPU virtualization flags from guest VMs for security and vMotion compatibility.

---

The Mess: The Locked 64-Bit VM and Silent Packet Drops

Our automated lab factory deployed 12 nested ESXi virtual machines (Pod01-ESXi-01, Pod01-ESXi-02).

The nested ESXi hosts installed cleanly and rendered the standard yellow-and-grey management console screen.

Then, our test engineer logged into the nested vCenter and attempted to power on a 64-bit Linux workload VM.

vCenter returned a hard blocking error:

# vSphere Client Error Log on Nested Workload Power-On
Failed to start the virtual machine: Pod01-Test-VM-01.
Error: Virtualized Intel VT-x/EPT is not supported on this platform.
CPUID: Disabling 64-bit guest support.

The nested VM refused to boot.

To make matters worse, even when the engineer created a 32-bit test VM that did boot, it couldn’t talk to any other VM on the nested network!

The 32-bit nested VM could ping its own local interface, but all traffic destined for other nested VMs on the same logical switch was silently dropped by the parent physical hypervisor.

# Diagnostic ping from nested VM to nested gateway
ping 10.200.1.1
# Output: Destination Host Unreachable (0 packets received)

The lab factory was completely dead in the water. We couldn’t run 64-bit workloads, and nested VMs couldn’t talk over the network.

---

The Solution: HV Passthrough & Promiscuous vDS Policies

We resolved both issues by making two mandatory configuration changes at the parent physical ESXi hypervisor layer:

1. Exposing Hardware-Assisted Virtualization (VT-x/EPT)

We updated the parent VM configuration (.vmx) for all nested ESXi virtual machines to pass physical CPU virtualization flags directly through to the guest OS.

# # Ansible Automation Task: Enabling Hardware HV Passthrough
- name: 'Enable Hardware VT-x/EPT Passthrough on Nested ESXi VM'
  community.vmware.vmware_guest:
    hostname: '{{ physical_esxi_host }}'
    username: '{{ sys_user }}'
    password: '{{ sys_pass }}'
    name: 'Pod01-ESXi-Node01'
    hardware:
      nested_virt: true # Exposes Intel VT-x / EPT flags
    customvalues:
      - key: 'vhv.enable'
        value: 'TRUE'

Setting nested_virt: true and vhv.enable = "TRUE" instructed the physical ESXi kernel to expose hardware VT-x/EPT CPU instructions to the nested hypervisor, allowing it to run 64-bit nested workload VMs seamlessly.

2. Overriding Physical vDS Security Policies

The reason nested VMs couldn’t communicate over the network was that the parent physical Distributed Switch (vDS) was enforcing security policies that blocked MAC address spoofing.

Because a nested ESXi host runs multiple nested VMs behind a single physical vNIC, the parent vDS sees traffic from multiple different MAC addresses coming from one virtual switch port.

By default, vDS drops this traffic as a security violation.

# Enabling Promiscuous Mode and Forged Transmits on Parent vDS Portgroup
Get-VDPortgroup "Nested-Lab-Trunk-PG" | Get-VDSecurityPolicy | Set-VDSecurityPolicy `
  -AllowPromiscuous $true `
  -ForgedTransmits $true `
  -MacChanges $true

# Parent vDS Security Policy Requirements for Nested ESXi

| Security Setting        | Default Physical Value | Required Nested Value | Reason                                                           |
| ----------------------- | ---------------------- | --------------------- | ---------------------------------------------------------------- |
| **Promiscuous Mode**    | Reject                 | **Accept**            | Allows parent vDS to forward frames for nested VM MACs           |
| **Forged Transmits**    | Reject                 | **Accept**            | Allows nested ESXi to transmit frames with nested VM source MACs |
| **MAC Address Changes** | Reject                 | **Accept**            | Allows nested ESXi vNICs to re-bind MAC addresses dynamically    |

---

The Impact

• 64-Bit Nested Support: Enabled 100% execution of 64-bit nested workload VMs across all automated SDDC lab pods.
• Complete Connectivity: Resolved silent packet drops, enabling multi-node nested vSAN and NSX overlay networking.
• Automated Lab Factory: Built an automated Ansible workflow that provisions complete, pre-configured nested SDDC environments in under 25 minutes.

Key Takeaway

Enable HV CPU Passthrough and Relax Parent vDS Security for Nested Hypervisors.

When building nested virtualization labs, always expose hardware VT-x/EPT CPU flags (vhv.enable = "TRUE") to guest VMs before powering them on. Additionally, set Promiscuous Mode, Forged Transmits, and MAC Address Changes to Accept on parent vDS portgroups so nested hypervisors can pass multi-MAC traffic across physical vNIC ports.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin