“Stretching a Layer-2 network across a WAN link is like laying a copper wire across an ocean. If someone causes a Spanning Tree loop in the physical datacenter, your cloud environment will crash three seconds later.”

In early 2020, during my tenure as a VMware NSX Consultant at IBM, we were tasked with migrating a major manufacturing client to IBM Cloud SDDC.

The application architecture consisted of a massive legacy ERP environment residing on a single stretched /22 subnet (1,024 host IP addresses). Re-IPing was off the table due to legacy license keys bound to static IP addresses.

The migration had to be phased over four months. That meant for 120 days, on-premise physical servers and newly migrated cloud VMs had to live on the exact same broadcast domain across a 10Gbps Direct Link.

---

The Fragile WAN Bridge

When you extend a Layer-2 broadcast domain over a WAN link, every ARP broadcast, NetBIOS announcement, and multicast packet generated on-premise travels over the IPsec L2VPN tunnel to the cloud.

Under normal conditions, this broadcast overhead consumes 5–10% of WAN bandwidth.

Under abnormal conditions, it destroys the link.

---

The Mess: The Unmanaged Switch That Crashed the Cloud

Three weeks into the migration, the entire hybrid cloud connectivity dropped.

The NOC paged in: IBM Cloud Direct Link down. All migrated cloud VMs unreachable.

The physical network team immediately blamed the fiber provider, assuming a physical fiber cut on the dedicated circuit between Delhi and the cloud datacenter.

I logged into the NSX Autonomous Edge L2VPN client appliance on-premise to check interface statistics. The fiber wasn’t cut. The tunnel was drowning.

# Checking interface packet counters on the NSX L2VPN Standalone Client
get edge-interface ge-0/1 stats
# Output: rx_packets: 4,500,000 | rx_broadcast: 4,200,000 | rx_pps: 92,000 pps (100% CPU saturation)

A junior technician in the local server room had plugged an unconfigured 8-port desktop switch into an open trunk port to test a staging server. He hadn’t enabled BPDU Guard or Spanning Tree on the switch.

A switching loop formed on-premise.

Within seconds, a broadcast storm of millions of looping ARP packets flooded the trunk, entered the NSX L2VPN client, and traversed the tunnel into the cloud. The NSX Edge CPU hit 100%, collapsed the IPsec tunnel state engine, and took down the entire hybrid bridge.

---

The Solution: Local ARP Proxy & Egress Optimization

We restored the link by shutting down the misconfigured switch port on-premise. But to prevent future loops from taking down the cloud, we implemented three strict structural safeguards inside the NSX L2VPN architecture.

1. Local ARP Proxy Suppression

We enabled Local ARP Proxy on the NSX L2VPN Autonomous Edge client. Instead of forwarding broadcast ARP requests across the WAN link to find a host, the NSX Edge intercepts the ARP request and responds locally using a cached IP-to-MAC mapping table.

# Enabling Local ARP Proxy on the NSX L2VPN Edge client via CLI
set service l2vpn-client arp-proxy enable
set service l2vpn-client broadcast-suppression threshold 500

2. Broadcast & Multicast Rate Limiting

We configured hard rate-limits on the L2VPN interface. Broadcast and multicast traffic was capped at 500 packets per second. Any broadcast surge above that threshold (such as a Spanning Tree loop) was dropped at the ingress interface before reaching the IPsec encryption pipeline.

3. Optimized Egress Routing (OER)

Without OER, a cloud VM trying to reach the internet would send traffic to its default gateway on-premise—hairpinning across the WAN link to the physical firewall and back again (the dreaded “trombone routing” effect).

# NSX-T Segment Egress Routing Optimization Block
resource "nsxt_policy_segment" "stretched_subnet" {
  display_name        = "Stretched_L2_Subnet_102"
  connectivity_path   = nsxt_policy_tier1_gateway.t1_gateway.path

  advanced_config {
    address_pooling   = true
    local_egress      = true  # Enables local default gateway routing per site
  }
}

By enabling Local Egress, cloud VMs routed internet and cross-subnet traffic directly out of the cloud Tier-0 gateway, eliminating WAN hair-pinning.

---

The Impact

• WAN Bandwidth Optimization: Reduced baseline WAN broadcast traffic by 80% using local ARP proxy caching.
• Resilience Against Loops: Broadcast rate-limiting successfully absorbed two subsequent physical switch loops on-premise without dropping the L2VPN tunnel.
• Latency Elimination: Local Egress Routing cut outbound web traffic latency for cloud VMs from 45ms (hairpinned) to 4ms (direct cloud egress).

Key Takeaway

Never Stretch Layer-2 Networks Without Broadcast Suppression.

Stretching subnets across sites should always be a temporary migration tactic. If you must stretch a VLAN, never deploy a plain Layer-2 bridge. You must enforce Local ARP Proxy, Broadcast Rate-Limiting, and Local Egress Routing to protect your cloud fabric from physical network failures on-premise.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin