The MTU 9000 Trap: Debugging Geneve Overlay Encapsulation Packet Drops

“In software-defined overlay networking, a physical MTU of 1500 bytes will silently drop encapsulated tenant jumbo frames every time.”

The Setup

In December 2018 at IBM, we designed multi-tenant Software-Defined Data Centers (SDDC) on IBM Cloud (SoftLayer) using VMware NSX-T. The overlay network relied on Geneve (Generic Network Virtualization Encapsulation) tunneling protocol to encapsulate tenant L2/L3 packets across physical ToR switches.

To support high-throughput database replication and storage vMotion, tenant virtual machines were configured with 9000 MTU jumbo frames.

The Mess

During database stress testing between ESXi transport nodes across different racks, database replication throughput collapsed from 10 Gbps to less than 40 Mbps.

Virtual machines experienced severe TCP retransmissions and ICMP packet drops whenever packet sizes exceeded 1450 bytes. Ping tests with DF (Don’t Fragment) bit set failed consistently:

[ESXI HOST VMKPING LOG] 2018-12-10 16:22:04 - ESXi Transport Node #04
# vmkping -I vmk10 -s 8950 -d 192.168.110.52
PING 192.168.110.52 (192.168.110.52): 8950 data bytes
sendto() failed (Message too long / Fragmented packet dropped by Geneve TEP)
Packet Loss Rate: 100.0%

The wrong initial diagnosis blamed defective physical NIC SFP+ transceivers.

In reality, physical switch interfaces were configured for standard MTU 9000. However, Geneve encapsulation adds 50 bytes of variable TLV option headers to every packet (IP + UDP + Geneve Header). When an ESXi host attempted to send a 9000-byte tenant payload, the encapsulated Geneve frame reached 9050 bytes, exceeding the physical switch MTU of 9000 bytes and triggering silent drops!

The Solution

I re-engineered the physical network fabric and vSphere Distributed Switch (vDS) MTU parameters to accommodate Geneve encapsulation overhead:

1. Physical Fabric MTU Expansion: Configured all physical ToR switches, leaf-spine links, and ESXi physical NICs to MTU 9600.
2. vSphere TEP MTU Tuning: Hardcoded ESXi TEP (Tunnel End Point) vmkernel interfaces to MTU 9000.
3. MSS Clamping Validation: Verified that internal tenant virtual machine TCP Maximum Segment Size (MSS) clamped correctly at 8960 bytes.

# ESXi Transport Node Post-Fix Verification
# esxcli network ip interface set -m 9000 -i vmk10
# vmkping -I vmk10 -s 8950 -d 192.168.110.52
PING 192.168.110.52 (192.168.110.52): 8950 data bytes
8958 bytes from 192.168.110.52: icmp_seq=0 ttl=64 time=0.412 ms
8958 bytes from 192.168.110.52: icmp_seq=1 ttl=64 time=0.398 ms
--- 192.168.110.52 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss

Key Takeaway

Always configure your physical network fabric MTU to at least 9600 bytes when deploying Geneve or VXLAN overlays. Accounting for the 50-byte Geneve encapsulation header prevents silent packet fragmentation and throughput degradation.

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin