“When dynamic BGP routes bounce from an SDN Tier-0 gateway to a physical core switch and back into a secondary gateway, you create a Transitive Routing Loop. Your physical switch CPU hits 100%, and half your cloud traffic dies in a TTL expiry death spiral.”

In March 2020, during my tenure as a VMware NSX Consultant at IBM Cloud, we were interconnecting an enterprise software-defined datacenter with a physical core network.

The topology featured two redundant NSX-T Tier-0 Edge Gateways (Edge-01 in Datacenter A, Edge-02 in Datacenter B) connected to two physical Cisco Nexus 7000 core switches via dynamic BGP peering.

The goal was high availability: if Datacenter A suffered a fiber cut, Edge-02 in Datacenter B would take over routing for all 300 tenant subnets automatically.

Instead of high availability, we created a self-inflicted denial-of-service attack within 60 seconds of turning on BGP.

---

The Transitive Feedback Loop

When you run dynamic routing protocols (BGP or OSPF) across both software-defined overlays and physical underlay networks, you must control Transitive Route Propagation.

Without explicit route filtering, BGP will do what BGP is programmed to do: propagate valid routes to all reachable neighbors.

Here is how our topology created an invisible feedback loop:

1. Edge-01 advertised overlay subnet 10.200.0.0/16 to Physical Core A via BGP.
2. Physical Core A accepted the route and advertised it across the physical Inter-Switch Link (ISL) to Physical Core B.
3. Physical Core B saw a valid BGP advertisement for 10.200.0.0/16 and advertised it downstream to Edge-02.
4. Edge-02 accepted the route from Core B and redistributed it back into the NSX-T Tier-0 management plane!

Now, Edge-01 believed the shortest path to its own overlay subnet was via Edge-02, which sent it to Core B, which sent it to Core A, which sent it back to Edge-01.

A Transitive Routing Loop was born.

---

The Mess: The 50% Packet Drop Death Spiral

Within one minute of bringing up the BGP adjacencies, the physical Cisco Nexus 7000 core switch CPUs hit 100% utilization.

# Physical Cisco Nexus Core Switch BGP Log
%BGP-5-ADJCHANGE: BGP route 10.200.0.0/16 flapping between NextHop 10.100.1.1 and 10.100.2.1
# Result: 50% of cross-datacenter packets dropped due to IP Time-To-Live (TTL) Expiry!

Packets entering the datacenter were trapped in an infinite loop between the physical switches and the virtual edge nodes until their IP Time-To-Live (TTL) counter reached zero.

Stateful firewalls situated on the transit link dropped 100% of returning sessions because outbound packets left via Edge-01 but returning packets re-entered via Edge-02, creating asymmetric firewall resets.

A junior network engineer tried to break the loop by putting a static /16 null-route on Core B.

It stopped the loop—and simultaneously broke failover. When we tested shutting down Edge-01, Core B’s static null-route prevented traffic from reaching Edge-02, creating a complete black-hole outage during failure testing!

Static routes could not handle dynamic failover. We had to fix the BGP control plane.

---

The Solution: BGP Community Tagging & Filtering

We removed the static null-routes and implemented a strict BGP Community Route-Tagging Policy across both Tier-0 Edge gateways and the physical Nexus core switches.

We assigned a unique BGP Community tag (65000:100) to all SDN overlay routes advertised by the Tier-0 gateways.

// # NSX-T Tier-0 Gateway Route-Map Configuration via REST API
{
  "display_name": "TAG-SDN-OVERLAY-ROUTES",
  "entries": [
    {
      "action": "PERMIT",
      "set": {
        "community": ["65000:100"]
      }
    }
  ]
}

On the physical Cisco Nexus core switches, we configured a BGP route map that inspected incoming routes. If a route carried the 65000:100 community tag, the switch permitted the route locally but set the BGP attribute to NO-ADVERTISE.

# Cisco Nexus BGP Route-Map Policy
ip bgp-community new-format
!
route_map MAP-BLOCK-TRANSITIVE-OVERLAY permit 10
 match community COMMUNITY-SDN-OVERLAY
 set community no-advertise additive
!
ip community-list standard COMMUNITY-SDN-OVERLAY permit 65000:100

Because the NO-ADVERTISE community tag instructed Core A not to re-advertise the overlay route to Core B (or to Edge-02), the transitive feedback loop was broken permanently.

If Edge-01 failed, Edge-02 began advertising 10.200.0.0/16 with tag 65000:100. Core B accepted the route, routed traffic directly to Edge-02, and maintained full high-availability failover in under 2 seconds.

---

The Impact

• Zero Transitive Loops: Permanently eliminated BGP feedback loops between software-defined overlay gateways and physical core networks.
• Sub-2-Second Failover: Delivered automated active/standby failover between Datacenter A and Datacenter B without human intervention.
• CPU Relief: Dropped physical core switch CPU utilization from 100% back to a baseline 12%.

Key Takeaway

Tag and Filter All Cross-Boundary BGP Advertisements.

Never redistribute dynamic routes between software-defined overlay gateways and physical core networks without applying explicit BGP Community Tags. Use NO-ADVERTISE community attributes to prevent physical core networks from re-advertising SDN overlay routes back into secondary gateways, protecting your fabric from transitive routing loops.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin