“If a network change isn’t represented by a Git commit, it doesn’t exist. Moving from manual SSH terminal sessions to a NetDevOps GitOps model isn’t just a tooling upgrade—it’s an organizational culture shift.”

In mid-2021, during my time as a Systems Integration Advisor at NTT Data, our network operations team was operating like a 1990s sysadmin shop.

Whenever a new VLAN needed to be provisioned or an ACL updated, an engineer opened Putty or SecureCRT, SSHed directly into the core switch or firewall CLI, entered configure terminal, and typed commands live in production.

There was zero version control, zero peer code review, and zero automated testing.

If an engineer made a typo, production broke immediately.

---

The Fatal Syntax Typo

Command Line Interfaces (CLIs) are unforgiving environments.

A single missing keyword or misplaced space in a Cisco IOS or Arista EOS command can completely alter the command’s behavior.

We had built a culture where senior engineers bragged about their ability to type CLI commands from memory under pressure.

It was a recipe for catastrophe.

---

The Mess: The Missing “Add” Keyword Outage

During a routine maintenance window at 2:00 AM, a senior network engineer logged into a core aggregation switch to add a new application VLAN (VLAN 200) to an existing trunk port channel (Port-Channel10).

He intended to type:

# What the engineer INTENDED to type:
interface Port-Channel10
 switchport trunk allowed vlan add 200

Under pressure, he omitted a single word: add.

# What the engineer ACCIDENTALLY typed:
interface Port-Channel10
 switchport trunk allowed vlan 200

In Cisco IOS syntax, omitting the add keyword does not append the new VLAN to the trunk—it overwrites the entire allowed VLAN list with only VLAN 200!

Instantly, VLANs 10, 20, 50, 100, and 150 were stripped from the core trunk link.

Twelve production subnets—including online customer ordering and inventory management—went completely dark.

# Core Switch Console Output following the fatal typo:
%ETHPORT-5-IF_DOWN_VLAN_REMOVED: Interface Port-Channel10, VLANs 10,20,50,100,150 removed from trunk

When the incident manager asked the engineer to revert the change immediately, panic set in. The engineer couldn’t remember the exact list of VLAN IDs that had been allowed on Port-Channel10 before he ran the command!

Because there was no version control or pre-change snapshot, the team spent three hours digging through stale text files and manual Excel spreadsheets to reconstruct the switch port configuration.

---

The Solution: The NetDevOps GitOps Architecture

We revoked direct SSH configure terminal access for all network engineers and mandated a NetDevOps GitOps Pipeline.

All network intent—VLAN allocations, BGP neighbor definitions, interface descriptions, and ACLs—was stored as YAML state files in a centralized Git repository.

# # NetDevOps Single Source of Truth (datacenter_inventory.yml)
all:
  children:
    core_switches:
      hosts:
        core-sw01.corp.local:
          ansible_host: 10.100.1.11
          port_channels:
            - name: 'Port-Channel10'
              description: 'Trunk to App Cluster A'
              allowed_vlans: [10, 20, 50, 100, 150, 200]

The 3-Stage GitOps Pipeline

# NetDevOps Pipeline Workflow

1. **Pull Request (PR):** Engineer submits a PR modifying `datacenter_inventory.yml` to add `VLAN 200`.
2. **Automated CI Validation:** GitLab CI runs YAML syntax checks, IP collision detection, and `ansible-playbook --check` dry-runs.
3. **Automated CD Deployment:** Upon peer review approval, Ansible Tower pushes the configuration to the target switch using declarative templates.

# Clean, idempotent Jinja2 template for switchport trunk management
interface {{ item.value.name }}
description {{ item.value.description }}
switchport mode trunk
switchport trunk allowed vlan {{ item.value.allowed_vlans | join(',') }}

Because the Jinja2 template explicitly formats the join(',') array, human syntax typos like omitting the add keyword are mathematically impossible.

If a deployment ever fails, rolling back is as simple as executing git revert on the merge commit—Ansible re-applies the previous known-good state in under 60 seconds.

---

The Impact

• Zero Syntax Typo Outages: Eliminated 100% of human CLI syntax typos across all production switches and firewalls.
• Complete Audit Trail: Every network change is cryptographically tracked in Git commit history with author attribution and peer approval logs.
• Rapid Rollback: Reduced incident recovery time for failed network changes from 3 hours (manual reconstruction) to 60 seconds (git revert).

Key Takeaway

Git Is the Single Source of Truth for Network State.

Never allow manual, unmonitored SSH terminal edits on production network hardware. Treat network configurations as software code. Implement a NetDevOps GitOps model using version-controlled Git repositories, automated CI validation pipelines, and declarative Ansible templates to guarantee configuration consistency.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin