RANCID & Subversion: How Automated Configuration Backups Saved a Core Router

“If your router configurations are not in automated version control, a single mistyped ‘write erase’ command will turn a 5-minute change into a 12-hour outage.”

The Setup

In January 2011 at Spectranet, we managed hundreds of Cisco 7206VXR core routers, 7606 PE routers, and Catalyst switches across regional POPs. Network changes occurred daily to turn up client BGP sessions and VLAN trunks.

We implemented open-source RANCID (Really Awesome New Cisco config Differ) combined with Subversion (SVN) to automatically log into every network device hourly, extract running configurations, and commit diffs to version control.

The Mess

During a Friday afternoon maintenance window, an engineer attempted to clear a single BGP neighbor session on a core Cisco 7206VXR router. Intending to run clear ip bgp 202.54.10.1, a copy-paste error executed a destructive script block:

Core-7206VXR# config t
Core-7206VXR(config)# no router bgp 45120
Core-7206VXR(config)# end
Core-7206VXR# write memory
Building configuration...
[OK] - 42 BGP Peering Sessions Destroyed.
%BGP-5-ADJCHANGE: neighbor 202.54.10.1 Down - BGP AS Reset
%SYS-5-CONFIG_I: Configured from console by admin_engineer on vty0 (192.168.100.50)

The entire BGP routing table collapsed. Over 1,500 active routes disappeared, severing transit connectivity for 40 enterprise clients.

The engineer realized with horror that no manual backup text file had been saved prior to executing the change.

The Solution

Because RANCID ran automated hourly diffs, we recovered the complete golden configuration in under 3 minutes:

1. RANCID SVN Diff Extraction: Queried the RANCID Subversion repository for the last known good commit prior to the change window.
2. Diff Verification: Isolated the exact 200-line BGP stanza from the SVN history.
3. Automated Re-injection: Pushed the golden BGP stanza back to the Cisco 7206VXR via SSH, restoring all 42 BGP sessions instantly.

# Extract Last Known Good BGP Configuration from RANCID SVN
svn diff -r 412:413 http://rancid.corp.local/svn/configs/core-7206vxr.corp.local

# RANCID Output - Golden BGP Stanza Restored
+router bgp 45120
+ bgp router-id 202.54.10.1
+ bgp log-neighbor-changes
+ neighbor 202.54.10.1 remote-as 174
+ neighbor 202.54.10.1 route-map PREPEND_OUT out
+ neighbor 202.54.10.5 remote-as 3549
+ neighbor 202.54.10.5 route-map LOCAL_PREF_IN in

The Results

Deploying RANCID version control provided immediate disaster recovery capabilities:

• MTTR (Mean Time to Recovery): Reduced from 12 hours (manual re-creation) to 3 minutes.
• Route Table Restoration: Restored 1,500 routes across 42 BGP sessions automatically.
• Audit Compliance: 100% of network CLI changes logged with timestamps and author IPs.

Key Takeaway

Never execute network changes without automated configuration version control running in the background. Deploying RANCID ensures every CLI change is versioned, diffed, and recoverable within minutes.

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin