“Stateful firewalls hate asymmetry. If your outbound packet leaves via ExpressRoute in Mumbai and the reply returns via Chennai, your Palo Alto NVA will silently drop the packet as an invalid TCP ACK—and your monitoring will blame the WAN link.”

In mid-2024, in my current role as Associate Director, Cloud Architecture & AI, I was called in to troubleshoot a multi-region Azure enterprise landing zone.

The client had dual ExpressRoute circuits: a primary 10Gbps link from Mumbai to Azure India Central, and a secondary 1Gbps link from Chennai to Azure India South. Inside Azure, traffic routed through a hub-and-spoke VNet topology protected by stateful Palo Alto Next-Gen Virtual Appliances (NVAs).

On paper, the dual ExpressRoute setup was built for high availability. In reality, it was dropping 15% of inter-region database connections every day.

---

The Asymmetric Black Hole

Stateful firewalls track the full lifecycle of a TCP connection. They expect to see the SYN, the SYN-ACK, and the final ACK pass through the exact same physical or virtual interface.

If outbound traffic passes through NVA-A in Mumbai, but the return traffic routes through NVA-B in Chennai, NVA-B sees a TCP ACK packet for a session it has no record of in its state table.

NVA-B does what stateful firewalls are programmed to do: it drops the packet silently as a potential TCP session-hijacking attempt.

To the application team, the connection simply hung. The database logs recorded a generic “TCP Connection Reset” error.

---

The Mess: The Static Route Loop Disaster

Before our architecture review, a junior engineer tried to resolve the drops by adding static /32 host routes to the Azure Virtual Network Route Tables (UDRs).

He added 40 static route entries pointing all subnets to the Mumbai NVA IP.

It worked for two hours—until the primary ExpressRoute circuit in Mumbai underwent scheduled provider maintenance.

Because the static routes were hardcoded, Azure continued routing outbound traffic to the Mumbai NVA, which attempted to send traffic over the dead Mumbai circuit. Meanwhile, incoming traffic from the client’s secondary datacenter was arriving via the Chennai ExpressRoute link.

Traffic looped infinitely between the hub virtual networks until the NVA interface packet buffers overflowed.

# How we diagnosed the static route loop inside the Azure Network Watcher
az network watcher show-next-hop --resource-group RG-NET --vm VM-DB-01 --source-ip 10.100.1.4 --dest-ip 10.200.5.10
# Output: Next Hop Type: VirtualAppliance | Next Hop IP: 10.100.0.4 (Mumbai NVA - Unreachable)

14 production spoke VNets went dark simultaneously. The quick fix had created a total outage.

flowchart TD
    subgraph OnPrem ["On-Premise Data Centers"]
        DC_Mumbai["Data Center A (Mumbai)\nCE Router 01"]
        DC_Chennai["Data Center B (Chennai)\nCE Router 02"]
    end

    subgraph Azure ["Azure Multi-Region Footprint"]
        ExpressRoute1["ExpressRoute Primary (Mumbai)\nLocalPref 200"]
        ExpressRoute2["ExpressRoute Secondary (Chennai)\nAS-Path Prepend x3"]
        ARS["Azure Route Server / Hub NVA"]
        Spokes["14 Spoke VNets (Prod & Shared)"]
    end

    DC_Mumbai <-->|Primary Path| ExpressRoute1
    DC_Chennai <-->|Failover Backup| ExpressRoute2

    ExpressRoute1 <--> ARS
    ExpressRoute2 <--> ARS
    ARS <--> Spokes

    style ExpressRoute1 fill:#0f172a,stroke:#34d399,stroke-width:2px;
    style ExpressRoute2 fill:#0f172a,stroke:#f59e0b,stroke-width:2px;
    style ARS fill:#1e293b,stroke:#38bdf8,stroke-width:2px;

---

The Solution: Deterministic BGP Path Manipulation

We removed all static UDR overrides and fixed the root cause at the routing protocol layer using Border Gateway Protocol (BGP) attribute tuning.

We engineered a deterministic BGP path selection policy across both ExpressRoute customer edge (CE) routers and Azure Route Server.

1. Inbound Traffic Steering (On-Premise to Azure)

We configured our on-premise BGP routers to prefer the Mumbai ExpressRoute path by setting Local Preference = 200 for all routes received from Azure India Central. The secondary Chennai path was assigned a lower Local Preference = 100.

2. Outbound Traffic Steering (Azure to On-Premise)

To force Azure to prefer the Mumbai path for outbound traffic, we configured AS-Path Prepending on the Chennai CE router. We prepended our Autonomous System Number (ASN) three times (AS 65000 65000 65000) on the Chennai BGP advertisement.

# BGP Path Prepending Configuration on Customer Edge Router (Cisco IOS)
router bgp 65000
 neighbor 169.254.1.1 remote-as 12076
 neighbor 169.254.1.1 route-map MAP-CHENNAI-OUT out
!
route-map MAP-CHENNAI-OUT permit 10
 set as-path prepend 65000 65000 65000

Azure’s internal BGP router evaluated the paths, saw the shorter AS-Path on the Mumbai circuit, and selected Mumbai for 100% of outbound traffic.

# Terraform configuration snippet for Azure ExpressRoute Circuit Peering
resource "azurerm_express_route_circuit_peering" "private_peering" {
  peering_type                  = "AzurePrivatePeering"
  express_route_circuit_name    = azurerm_express_route_circuit.er_primary.name
  resource_group_name           = azurerm_resource_group.rg_network.name
  peer_asn                      = 65000
  primary_peer_address_prefix   = "10.0.0.0/30"
  secondary_peer_address_prefix = "10.0.0.4/30"
  vlan_id                       = 100
}

If the Mumbai circuit drops, BGP automatically withdraws the primary routes. Azure seamlessly fails over to the Chennai path within sub-seconds—and because both outbound and inbound traffic shift to Chennai simultaneously, asymmetric routing drops remain at zero.

---

The Impact

• Zero Asymmetric Drops: Eliminates 100% of TCP session drops caused by firewall state mismatches across multi-region circuits.
• Automated Failover: BGP route withdrawal enables automated failover between Mumbai and Chennai in under 3 seconds without human intervention.
• Static Route Removal: Stripped 40 static /32 UDR overrides, returning VNet routing control to dynamic BGP propagation.

Key Takeaway

Architect ExpressRoute Links for Symmetric BGP Resilience.

Never rely on static route overrides (UDRs) to control multi-region cloud WAN paths. Use native BGP attributes—Local Preference for inbound traffic and AS-Path Prepending for outbound traffic—to ensure stateful NVAs see symmetric bidirectional traffic flows.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin