Azure ExpressRoute BGP Route Weight Tuning & Active-Active WAN Steering

“Active-Active WAN links require per-prefix BGP MED steering — active-passive wastes half your provisioned cloud bandwidth.”

The Setup

In February 2024 in my current role as Associate Director, we managed enterprise cloud connectivity scaling across two 10Gbps Azure ExpressRoute circuits. Running an Active-Passive topology left 10Gbps of expensive secondary bandwidth completely idle during normal operations.

We wanted to transition to an Active-Active WAN steering model where Production Subnet A used ExpressRoute 1, while Production Subnet B used ExpressRoute 2, with automatic sub-second failover between both paths.

The Mess

Simply advertising both subnets equally over both ExpressRoute circuits caused ECMP (Equal-Cost Multi-Path) hash instability.

Stateful Palo Alto firewalls at the cloud perimeter began dropping 15% of active TCP flows due to out-of-order packet arrival:

[WARNING] 2024-02-12 14:10:00 UTC - Azure Gateway Connection Log
ExpressRoute Gateway: er-gw-prod-01
Circuit 1 Utilization: 9.2 Gbps (CONGESTED - 92%)
Circuit 2 Utilization: 0.4 Gbps (UNDERUTILIZED - 4%)
BGP Flaps: 12 BGP Route Flaps detected on MSEE Peer 192.168.250.2

Because Azure ExpressRoute gateways by default select paths based on internal BGP Weight, all return traffic from Azure converged on Circuit 1, creating severe link congestion while Circuit 2 remained underutilized.

The Solution

I designed per-prefix Multi-Exit Discriminator (MED) and Azure Connection Weight tuning to create symmetric, active-active traffic steering:

1. On-Premises Egress Steering via BGP MED: Advertised Subnet A with MED 100 on Circuit 1 and MED 200 on Circuit 2. Advertised Subnet B with MED 200 on Circuit 1 and MED 100 on Circuit 2.
2. Azure Return Egress Steering via Connection Weight: Set Azure Connection Weight to 200 on ER-Connection-1 for Subnet A VNet peerings, and 200 on ER-Connection-2 for Subnet B VNet peerings.

# Azure CLI - Set ExpressRoute Connection Weight for Active-Active Steering
az network vpn-connection update \
  --name "conn-expressroute-primary" \
  --resource-group "rg-networking-prod" \
  --routing-weight 200

az network vpn-connection update \
  --name "conn-expressroute-secondary" \
  --resource-group "rg-networking-prod" \
  --routing-weight 100

Key Takeaway

To achieve true Active-Active ExpressRoute load balancing, combine per-prefix BGP MED advertisements from on-premises with Azure Routing Weight configuration at the virtual network gateway boundary.

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin