“The vendor migration guide made it look simple: click ‘Migrate’ in the Migration Coordinator, wait for the progress bar, and celebrate. What the guide omitted was what happens when the migration engine hangs at 68% with 200 production virtual machines split between two different control planes.”

In early 2020, during my time as a VMware NSX Consultant at IBM, a major financial services client faced a hard lifecycle deadline. Their NSX-V environment was going end-of-support.

With ₹0 allocated for new hardware, we couldn’t build a new NSX-T cluster and migrate workloads side-by-side. We had to perform an in-place migration on the existing production ESXi hosts.

---

The In-Place Architecture

An in-place migration requires the hypervisor to swap its networking kernel modules on the fly.

The host must transition from the legacy NSX-V kernel modules (vdl2) to the modern NSX-T modules (nsx-vdl2) while virtual machines are running. The migration Coordinator tool is supposed to automate this step-by-step.

But automating a control plane swap on live database hosts is like swapping engines on a jet mid-flight.

---

The Mess: The 68% Hang

During the host migration phase on the primary database cluster, the progress bar stopped moving.

68% complete.

Ten minutes turned into thirty. The migration coordinator log was flooded with timeout errors.

Half of our ESXi hosts in the cluster had upgraded to the NSX-T transport zone, but the remaining hosts were still running NSX-V. The host cluster was split down the middle.

# Checking host vib status on the stuck hypervisors
esxcli software vib list | grep -E "vmware-esx-nsxv|nsx-ops"
# Output: Host-03: Running legacy vdl2 kernel modules
# Host-04: Running new nsx-vdl2 kernel modules

Workloads that vMotioned from Host-03 to Host-04 immediately lost network connectivity. The control planes could not sync the logical switch state.

To make matters worse, the policy translation engine had failed to convert the legacy vCenter-based Security Tags. Instead of translating them into NSX-T group criteria, the engine ignored them, leaving the migrated VM ports with no rules applied—meaning they hit the default-deny policy.

Active-active database sync broke. The backup cluster went offline.

graph LR
    subgraph NSX_V ["Legacy NSX-V Environment (VXLAN)"]
        HostV3["ESXi Host 03\nvdl2 kernel module"]
        VMV["Database VM (NSX-V Tag)"]
        HostV3 --- VMV
    end

    subgraph HCX ["HCX L2 Stretch Bridge"]
        L2VPN["L2VPN Concentrator / HCX Monitored Transit"]
    end

    subgraph NSX_T ["Target NSX-T Environment (GENEVE)"]
        HostT4["ESXi Host 04\nnsx-vdl2 kernel module"]
        VMT["Migrated VM (NSX-T Group)"]
        HostT4 --- VMT
    end

    NSX_V <-->|HCX Bulk Live vMotion| HCX
    HCX <-->|Zero IP Change| NSX_T

    style NSX_V fill:#0f172a,stroke:#ef4444,stroke-width:2px;
    style HCX fill:#0f172a,stroke:#38bdf8,stroke-width:2px;
    style NSX_T fill:#0f172a,stroke:#34d399,stroke-width:2px;

---

The Solution: Control Plane Recovery & Scripted Group Translation

We had to bypass the broken automation coordinator and resolve the state mismatch manually.

First, we isolated the control plane split by temporarily disabling DRS (Distributed Resource Scheduler) to stop VMs from vMotioning between the mismatched hosts.

Second, I wrote a Python script to connect to the NSX-V Manager API, extract all Security Tags, and manually inject them as tags into the new NSX-T Policy API.

# REST API call sequence to sync legacy tags to NSX-T policy objects
import requests

def sync_tag_to_nsxt(vm_id, tag_name):
    url = f"https://nsx-manager.local/api/v1/fabric/virtual-machines/{vm_id}/tags"
    payload = {
        "tags": [{"scope": "security_tag", "tag": tag_name}]
    }
    headers = {"Content-Type": "application/json"}
    # Apply tag to VM object in NSX-T inventory database
    response = requests.put(url, json=payload, headers=headers, auth=('admin', 'password'), verify=False)
    return response.status_code

We manually forced the remaining hosts to complete the kernel module upgrade via command line, skipping the coordinator’s failed orchestration loops.

# Forcing the install of the new NSX-T kernel module VIB on the command line
esxcli software vib install -v /var/log/vmware-esx-nsx-ops.vib --force

Once all hosts were running the same kernel modules, we enabled the dynamic security groups based on the tags we had just synced. The database sync immediately recovered.

---

The Impact

• Cluster Recovery: Restored the split control plane to a uniform state without having to reboot the ESXi hosts or drop VM power states.
• DFW Policy Preserved: Successfully translated all 450 security tags to the new NSX-T policy model.
• Zero Data Loss: Database transactions recovered without losing a single database record during the recovery window.

Key Takeaway

Never Trust a Closed-Source Migration Wizard Without a Manual Rollback Plan.

When performing in-place platform upgrades, always document the raw API payloads and CLI commands required to finish the job manually. If the automation engine hangs, you must be prepared to step in at the kernel level to align the control plane state.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin