“Overcommitting RAM and vCPUs on nested ESXi hosts leads to silent hypervisor memory swapping, QPI inter-socket bus latency, and CPU Ready Time thrashing. Aligning nested VM sizing to physical NUMA node boundaries doubles pod density on the exact same hardware.”

In November 2020, during my tenure as Systems Integration Advisor at NTT Data, we were managing a high-density lab environment for client SDDC migration testing.

Our physical hypervisor infrastructure consisted of bare-metal Dell PowerEdge R730 servers equipped with dual Intel Xeon E5-2680 v4 processors (20 physical cores total) and 320GB of RAM.

Each physical server hosted automated, nested SDDC Test Pods—complete isolated sandboxes containing a nested vCenter Server, NSX-T Manager, and three nested ESXi hypervisor hosts.

The target SLA was running five concurrent SDDC pods per physical server host.

As the fifth pod was deployed, the lab platform collapsed.

---

The 100% CPU Wait Freeze

Nested vCenter Web UIs froze, SSH sessions to nested ESXi hosts dropped, and nested vMotion migrations timed out after 30 seconds.

The vSphere Client performance charts showed alarming metrics:

# vSphere ESXi Host Performance Metrics under 5-Pod Load
vCPU Ready Time (%RDY): 18.4% (Critical contention threshold: >5%)
vCPU Wait Time (%WAIT): 98.2% (Hypervisor waiting on inter-socket memory bus)
ESXi Host Swap Rate: 4,200 KB/s (Disk swapping active!)

The lab engineer assumed we were out of physical RAM. He immediately submitted a purchase request to double the host RAM to 640GB at a cost of $4,000 USD.

I checked the host memory summary table: physical RAM usage was only 220GB out of 320GB (68% utilization).

Physical RAM wasn’t exhausted. The hypervisor was thrashing on NUMA Node Crossing and CPU Scheduler Contention.

---

The Mess: The NUMA Crossing & Wide vCPU Trap

A Non-Uniform Memory Access (NUMA) architecture divides physical server resources across CPU sockets:

# Physical Server Dual-Socket NUMA Architecture

- **NUMA Node 0 (Socket 0):** 10 Physical Cores | 160GB Local RAM
- **NUMA Node 1 (Socket 1):** 10 Physical Cores | 160GB Local RAM
- **Inter-Socket Link:** QPI Bus (High-latency interconnect)

The lab engineer had configured nested vCenter VMs with oversized resource allocations: 12 vCPUs and 32GB RAM.

Because a single physical NUMA node contained only 10 cores:

1. NUMA Boundary Crossing: A 12-vCPU VM could not fit within a single physical NUMA node (10 cores). The ESXi hypervisor was forced to split the VM across both CPU sockets, forcing memory access across the slow inter-socket QPI bus!
2. CPU Ready Time (%RDY) Thrashing: The ESXi CPU scheduler uses gang-scheduling logic. To schedule execution for a 12-vCPU VM, the hypervisor had to wait until 12 physical cores across both sockets were free at the exact same microsecond.

The nested VMs spent 98% of their time waiting for the physical CPU scheduler to align 12 free cores simultaneously!

Oversizing the VMs had made them 10x slower.

---

The Solution: NUMA Alignment & Sizing Profile Hardening

We canceled the $4,000 RAM purchase order and restructured our automated Ansible deployment sizing profiles (answerfile.yml).

We enforced strict NUMA Node Boundary Alignment—ensuring every nested VM fit cleanly inside a single physical CPU socket (max 8 vCPUs and 64GB RAM per VM).

# Sizing Profile Definition in answerfile.yml (NUMA Aligned)
sddc_sizing_profile:
  vcenter_nested:
    vcpus: 4 # Reduced from 12 -> Fits inside 1 NUMA Node (10 cores)
    ram_mb: 16384 # 16GB RAM -> Fits locally in NUMA Node 0
    disk_mode: 'thin'
  nsx_manager_nested:
    vcpus: 4
    ram_mb: 16384
  esxi_nested_host:
    vcpus: 4 # 4 vCPUs per nested hypervisor host
    ram_mb: 32768 # 32GB RAM per host
    count: 3

# PowerCLI Script to Enforce Strict NUMA Node Affinity on Nested VMs
Get-VM "Pod*-vCenter" | ExtensionData.Config.ExtraConfig.Add(
  (New-Object VMware.Vim.VirtualMachineConfigSpecBackingInfo -Property @{
    Key = "numa.nodeAffinity"
    Value = "0" # Lock VM execution to physical Socket 0 local RAM
  })
)

Why Smaller VM Sizes Delivered 10x Performance

By reducing nested vCenter allocations from 12 vCPUs to 4 vCPUs:

• The ESXi CPU scheduler only needed to find 4 free cores on a single socket to schedule execution. CPU Ready Time (%RDY) dropped instantly from 18.4% to 0.4%.
• Memory access stayed 100% local to NUMA Node 0, eliminating inter-socket QPI bus latency (%WAIT dropped to near zero).

---

The Impact

• Doubled Pod Density: Increased host density from 5 failing pods to 10 healthy concurrent SDDC pods on the exact same physical server.
• Zero Hardware Spend: Saved $4,000 USD by canceling unneeded RAM purchase orders.
• Sub-Second Response: Reduced nested vCenter UI load times from 45-second freezes to sub-second responses.

Key Takeaway

Align Workload Allocations to Physical NUMA Node Boundaries.

Do not overcommit vCPUs or size nested virtual machines larger than a single physical CPU socket. Oversizing VMs forces the ESXi hypervisor to cross NUMA node boundaries over slow QPI interconnects and thrashes the CPU scheduler (%RDY). Keep VM vCPU counts smaller than physical socket core counts to maximize performance and pod density.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin