BGP Local-Pref Steering: Eliminating Asymmetric Routing Across Dual Azure ExpressRoute Circuits

“In dual-homed cloud WAN steering, asymmetric routing is a silent killer that drops stateful firewall sessions without warning.”

The Setup

In November 2023 in my current role as Associate Director, we architected high-availability connectivity for enterprise Azure Landing Zones using dual 10Gbps Azure ExpressRoute circuits (Primary in Region East, Secondary in Region West).

Both circuits established eBGP peering sessions with on-premises Customer Edge (CE) routers to advertise corporate subnets to the Microsoft Enterprise Edge (MSEE).

The Mess

During routine maintenance on the primary ExpressRoute circuit, on-premises firewalls began randomly dropping outbound application traffic to Azure virtual networks:

[ALERT] 2023-11-14 11:20:04 UTC - Palo Alto NGFW Log
Rule: Deny-Asymmetric-State-Violation
Source: 10.100.4.50 (On-Prem App Server) -> Destination: 10.200.12.10 (Azure VM)
Action: DROP (Reason: TCP SYN-ACK received on Secondary Circuit without matching SYN on session table)

The diagnosis revealed severe Asymmetric Routing:

• Outbound traffic from on-premises to Azure routed over ExpressRoute Primary because CE routers preferred local OSPF metrics.
• Inbound traffic returning from Azure routed over ExpressRoute Secondary because Microsoft MSEEs selected the lowest latency path dynamically.
• The stateful Palo Alto firewall on the secondary path received TCP ACK packets for sessions it never saw, dropping packets due to state violations.

The Solution

I engineered deterministic ingress and egress BGP steering across both ExpressRoute circuits:

1. Egress Steering via BGP Local-Pref: Applied local-preference 200 on CE routers for routes received via Primary ExpressRoute, ensuring all outbound traffic favors Circuit A.
2. Ingress Steering via AS-Path Prepending: Prepended our Autonomous System Number (ASN) 3 times (as-path prepend 65001 65001 65001) on Secondary ExpressRoute route advertisements to Microsoft MSEEs.

! Cisco CE Router BGP ExpressRoute Steering Configuration
router bgp 65001
 neighbor 192.168.254.1 remote-as 12076
 neighbor 192.168.254.1 route-map ER_PRIMARY_IN in
 neighbor 192.168.254.1 route-map ER_PRIMARY_OUT out
!
route-map ER_PRIMARY_IN permit 10
 set local-preference 200
!
route-map ER_SECONDARY_OUT permit 10
 set as-path prepend 65001 65001 65001

Key Takeaway

Always pair egress BGP Local-Preference tuning with ingress AS-Path Prepending when connecting multi-region Azure ExpressRoute circuits. Symmetric routing is mandatory when stateful firewalls inspect cloud boundary traffic.

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin