“Before modern Cloudflare Anycast DNS or AWS Route53 existed, steering global user traffic dynamically across physical data centers 1,000 miles apart required Cisco GSS DNS delegation, low TTLs, and real-time Keep-Alive Application Protocol (KALAP) health probes.”

In April 2012, during my tenure as Assistant Manager at Net4 India, we were building out a multi-region active-active hosting architecture for our enterprise clients.

The topology spanned two physical datacenters separated by 1,000 miles: Primary in Delhi and Secondary in Chennai.

Our goal was high availability: if Delhi suffered a fiber cut or datacenter outage, user traffic should steer automatically to the Chennai datacenter with zero human intervention.

Today, developers accomplish this with a single click in Cloudflare or AWS Route53.

In 2012, Cloudflare Anycast DNS didn’t exist for enterprise datacenters. We had to build Global Server Load Balancing (GSLB) using physical hardware appliances.

---

The 24-Hour DNS Cache Trap

The client’s internal application team initially attempted a manual DNS failover approach.

They configured static DNS A records at the domain registrar pointing app.net4.in directly to Delhi’s public Virtual IP (VIP: 202.71.130.100) with the registrar’s default Time-To-Live setting: 86,400 seconds (24 hours).

During a severe monsoon storm in April 2012, a fiber cut severed Delhi’s primary ISP backbones. The Delhi datacenter went dark.

The application team logged into the domain registrar console and manually updated the DNS A record to point to Chennai (VIP: 202.71.200.100).

They expected traffic to shift immediately.

Instead, end-users across India continued hitting Delhi’s dead IP address for 24 full hours.

Because the original DNS record had a 24-hour TTL, recursive DNS servers at major ISPs (BSNL, Airtel, Reliance) cached Delhi’s dead IP address. No matter how many times the team updated the registrar, client browsers refused to re-query DNS until the 24-hour cache expired!

The client lost 24 full hours of online e-commerce transactions.

Static DNS records with high TTLs were completely incapable of handling disaster recovery failovers.

---

The Solution: Dynamic GSLB with Cisco GSS 4400 & CSS 11500

We replaced static DNS records with a 2-tier hardware GSLB architecture deploying Cisco GSS 4400 (Global Site Selector) appliances and Cisco CSS 11500 (Content Services Switch) load balancers.

# 2-Tier Hardware GSLB Architecture

1. **DNS Delegation (GSS 4400):** Authoritative NS records for `app.net4.in` delegated to dual Cisco GSS appliances in Delhi and Chennai.
2. **KALAP Health Probes:** GSS appliances poll local Cisco CSS 11500 load balancers every 5 seconds via KALAP (Keep-Alive Application Protocol) and HTTP probes.
3. **Dynamic Low-TTL Answers:** GSS returns DNS `A` records with a **30-second TTL**, dynamically altering the IP response based on real-time VIP health.

# Cisco GSS 4400 Global Site Selector CLI Configuration
gss-config
  answer-alias APP-GSLB-VIP
    answer-type a-record
    add-answer 202.71.130.100  # Delhi VIP
    add-answer 202.71.200.100  # Chennai VIP

  keepalive KALAP-DELHI
    type kalap-css
    ip-address 202.71.130.1
    frequency 5

  dns-clause APP-ROUTING-RULE
    match-domain app.net4.in
    enable-answer APP-GSLB-VIP
    balance-method least-loaded

How Dynamic GSLB Handled Regional Failovers

1. Continuous Health Probing: GSS appliances sent KALAP and HTTP /healthcheck.php probes to local Cisco CSS load balancers in Delhi and Chennai every 5 seconds.
2. 30-Second TTL Enforcement: All DNS A record answers returned by GSS were hardcoded with TTL = 30 seconds.
3. Automated Sub-Minute Failover: If Delhi’s ISP links died or the CSS reported web server pool failure, GSS stopped returning 202.71.130.100. Within 30 seconds, ISP recursive DNS caches expired, and all new user requests were directed to Chennai (202.71.200.100).

---

The Impact

• Sub-Minute Failover: Reduced multi-datacenter failover times from 24 hours (static DNS) down to 30 seconds.
• Active-Active Load Distribution: Used GSS least-loaded balancing algorithms to distribute incoming traffic 60/40 between Delhi and Chennai based on real-time server CPU loads.
• Zero Human Intervention: Delivered fully automated disaster recovery steering without requiring manual NOC intervention during fiber outages.

Key Takeaway

Leverage Health-Aware Dynamic DNS Delegation for Multi-Region High Availability.

Never rely on static DNS A records with high TTLs for multi-datacenter disaster recovery. Delegate domain resolution to dynamic, health-aware GSLB engines (Cisco GSS/CSS, F5 BIG-IP GTM, or Route53/Cloudflare), enforce short 30-second DNS TTLs, and poll application health probes continuously to steer user traffic away from failed datacenters automatically.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin