“Migrating an enterprise core network from NSX-V to NSX-T without spare hardware is like swapping jet engines in mid-flight. When your 15-minute maintenance window hits minute 13 and your new BGP edges won’t peer, you find out what your team is really made of.”

In January 2021, during my time as a Systems Integration Advisor at NTT Data, VMware announced the end-of-support deadline for NSX-V.

Our client—a major regional financial institution—operated 400 production virtual machines hosting online banking and payment gateways on legacy NSX-V.

Because capital expenditure budgets were frozen for Q1, there was ₹0 allocated for spare hardware. We couldn’t build a new NSX-T cluster alongside the old environment and migrate virtual machines over time.

We had to execute an in-place migration directly on the live production hardware.

---

The North-South Routing Cutover

An in-place migration requires converting hypervisors and virtual networks in phases, but the most dangerous phase is the North-South Routing Cutover.

The legacy NSX-V environment used OSPF to peer Edge Services Gateways (ESGs) with the datacenter’s physical Cisco Nexus 7000 core switches.

The new NSX-T architecture replaced the ESGs with Tier-0 Gateways using BGP.

To execute the cutover, we scheduled a tight 15-minute maintenance window at 2:00 AM on a Sunday. The plan: shut down OSPF on the legacy ESGs, enable BGP on the new NSX-T Tier-0 edges, and let traffic re-route over the new GENEVE overlay.

---

The Mess: 90 Seconds to Rollback

At 2:00 AM, the cutover began.

# 2:01 AM - Tearing down legacy NSX-V OSPF neighbor adjacencies
nsx-esg01> no router ospf 100
# Result: OSPF adjacencies down. Traffic paused.

At 2:03 AM, we enabled BGP peering on the new NSX-T Tier-0 Gateways.

Nothing happened.

The BGP neighbor status remained stubbornly stuck in ACTIVE state—meaning the Tier-0 gateway was trying to establish a connection, but the physical Cisco Nexus core switches were refusing to complete the TCP handshake.

Online banking was dark.

By 2:10 AM—ten minutes into our 15-minute window—the client’s CTO on the bridge call spoke up: “You have three minutes before I order a full rollback to NSX-V.”

Rolling back meant two weeks of failed audit reviews and a high-risk rollback procedure on a partially-migrated hypervisor fabric.

We ran a real-time BGP diagnostic query directly on the Cisco Nexus CLI:

# Diagnostic query on the Cisco Nexus 7000 core switch
nexus7k-core01# show ip bgp neighbors 10.100.1.2 | grep -E "BGP state|Notification"
# Output: BGP state = Active
# Error: %BGP-3-NOTIFICATION: sent to neighbor 10.100.1.2 2/5 (Authentication Failure)

The smoking gun appeared instantly: Authentication Failure.

During pre-migration staging three weeks earlier, the physical network team had updated the BGP MD5 authentication password on the Nexus switch, but failed to document the password change in the migration answerfile. The new NSX-T Tier-0 gateway was trying to authenticate using an outdated pre-shared key!

---

The Solution: The Real-Time BGP Key Fix

With 90 seconds remaining on the CTO’s timer, we updated the MD5 secret on the NSX-T Tier-0 Gateway via REST API.

// # Emergency REST API update to fix BGP MD5 secret on Tier-0 Gateway
PUT https://nsx-mgr.local/policy/api/v1/infra/tier-0s/T0-GW-01/locale-services/default/bgp/neighbors/10.100.1.2
{
  "neighbor_address": "10.100.1.2",
  "remote_as": "65001",
  "password": "UpdatedSecureKey2021!"
}

Within two seconds of pushing the payload, the Nexus logs cleared:

nexus7k-core01# %BGP-5-ADJCHANGE: bgp-65000 [100] neighbor 10.100.1.2 Up

BGP adjacencies converged instantly. Route tables populated. At 2:14 AM—with 60 seconds to spare—the first synthetic online banking transaction completed successfully over the new NSX-T GENEVE overlay network.

---

The Impact

• Zero Rollback: Successfully completed the in-place NSX-V to NSX-T migration without triggering an emergency rollback.
• Modernized Encapsulation: Transitioned 400 VMs from legacy VXLAN to container-ready GENEVE encapsulation with variable-length TLV header support.
• Zero Downtime Execution: Preserved active application state across all core banking workloads outside the planned 15-minute edge cutover window.

Key Takeaway

Always Verify Layer-3 Authentication Keys In Real-Time Before Major Cutovers.

When migrating network control planes, never assume staging configurations match live physical switch settings. Always pull live routing diagnostic logs (show ip bgp neighbors) at the physical layer the moment an adjacency fails, rather than assuming hypervisor or CNI failure.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin