“Replacing your software-defined networking control plane while thousands of production VMs are actively communicating is like swapping jet engines mid-flight. If you don’t understand the difference between VXLAN and GENEVE headers, you will drop production packets.”

In late 2019, during my tenure as a VMware NSX Consultant at IBM Cloud, we led an enterprise SDN transformation.

Our client—a major multi-tenant cloud provider—was upgrading their core virtual network fabric from legacy NSX-V (bound to a single vCenter instance) to modern NSX-T (multi-hypervisor, container-ready, and autonomous).

The migration involved 300 ESXi hypervisors and thousands of tenant workloads processing financial and e-commerce transactions.

The constraint: zero application downtime allowed.

---

The Encapsulation Protocol Wall

The biggest technical hurdle in an NSX-V to NSX-T migration isn’t the management UI—it is the underlying packet encapsulation protocol.

• NSX-V Overlay: Uses VXLAN encapsulation (UDP destination port 4789) with a fixed 8-byte header.
• NSX-T Overlay: Uses GENEVE encapsulation (UDP destination port 6081) with variable-length Type-Length-Value (TLV) options for container context.

Because VXLAN and GENEVE use different UDP ports and different header formats, an ESXi host running NSX-V cannot process packets sent by an ESXi host running NSX-T over the physical underlay network.

They speak completely different wire protocols.

---

The Mess: The Split-Overlay Packet Black Hole

Before our architecture review, a junior deployment engineer attempted to upgrade a host cluster by applying the new NSX-T Transport Node Profile directly to a live ESXi host group.

He expected the host network interfaces to convert automatically.

Instead, the cluster split down the middle.

Three ESXi hosts converted to NSX-T (nsx-vdl2 driver, GENEVE port 6081), while the remaining three hosts remained on legacy NSX-V (vdl2 driver, VXLAN port 4789).

# Diagnostic output on the converted ESXi host (NSX-T GENEVE TTEP)
esxcli network ip interface ipv4 get -i vmk10
# Output: Interface vmk10 | IPv4: 192.168.100.50 | Type: GENEVE TTEP

# Diagnostic output on the unconverted host (NSX-V VXLAN VTEP)
esxcli network ip interface ipv4 get -i vmk02
# Output: Interface vmk02 | IPv4: 192.168.100.12 | Type: VXLAN VTEP

When a web application VM on Host-01 (GENEVE) sent a TCP request to a database VM on Host-04 (VXLAN), Host-01 encapsulated the frame into a GENEVE UDP 6081 packet.

Host-04 received the UDP 6081 packet, but because its kernel was only listening for VXLAN UDP 4789 packets, the network stack dropped the packet as unparseable junk data.

# Host-04 Kernel Log showing dropped unparseable UDP packets
vmkernel.log: Drop packet: Unknown UDP destination port 6081 on interface vmk02

150 tenant workloads lost internal network connectivity instantly.

---

The Solution: Dual-Stack L2 Bridging & Rolling Host Waves

We resolved the outage by temporarily placing the split hosts into maintenance mode, reverting the profile, and establishing a Dual-Stack Migration Architecture.

# Dual-Stack SDN Migration Staging Architecture

1. **Deploy NSX-T L2 Bridge:** Instantiate a dedicated NSX-T Layer-2 Bridge instance that stitches legacy VXLAN Logical Switches to new GENEVE Overlay Segments in hardware/VM memory.
2. **MTU 1600 Verification:** Expand physical switch fabric MTU to 1600+ bytes to accommodate GENEVE TLV header overhead.
3. **Rolling Host Maintenance Waves:** Place ESXi hosts in maintenance mode, vMotion workloads to legacy hosts, upgrade host kernel VIBs to NSX-T, and re-anchor workloads to GENEVE segments.

# Verifying MTU 1600 GENEVE ping connectivity between TTEP endpoints
ping ++Segment=vmk10 -s 1572 -d 192.168.100.51
# Output: 1572 bytes from 192.168.100.51: icmp_seq=1 ttl=64 time=0.412 ms (No Fragmentation)

By placing an NSX-T L2 Bridge between the legacy VXLAN switches and new GENEVE segments:

• VMs remaining on NSX-V hosts continued communicating over VXLAN.
• VMs migrated to upgraded NSX-T hosts communicated over GENEVE.
• The L2 Bridge translated packets between VXLAN (port 4789) and GENEVE (port 6081) in real-time, allowing cross-overlay communication during the 48-hour migration window without dropping a single TCP session.

---

The Impact

• Zero Downtime Migration: Successfully migrated 300 ESXi hypervisors and thousands of tenant workloads from NSX-V to NSX-T with zero application downtime.
• Packet Translation: Processed over 200 Million cross-overlay packets through the temporary L2 Bridge without packet corruption.
• Container Preparedness: Delivered GENEVE overlay capability with TLV header support, enabling future Kubernetes CNI integration.

Key Takeaway

Isolate Control Plane Migration from Data Plane Cutover.

Never attempt live SDN upgrades by applying new transport profiles to unisolated host clusters. Because VXLAN and GENEVE use incompatible wire formats, you must deploy temporary Layer-2 Bridges and verify MTU 1600+ underlay capabilities to translate overlay traffic dynamically during host migration waves.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin