“A 3,200-line single Ansible playbook is not automation—it’s technical debt waiting to explode. If changing a host network variable breaks your BGP peering task 2,000 lines lower down, your IaC architecture is broken.”

In early 2021, during my time as a Systems Integration Advisor at NTT Data, we were building zero-touch provisioning (ZTP) pipelines for private cloud environments.

The automation suite was supposed to spin up a complete Software-Defined Data Center (SDDC)—nested ESXi hypervisors, vCenter, NSX-T Manager clusters, and BGP routing—with zero human interaction.

It was an ambitious goal. But the codebase had become an unmaintainable nightmare.

---

The Monolithic Monster

Everything was packed into a single, massive Ansible file: deploy.yml.

At 3,200 lines, deploy.yml was impossible to reason about. Variable scoping was a free-for-all. The vCenter IP address was defined in fourteen different places under slightly different variable names (vcenter_ip, vCenter_Host, vc_address).

Running ansible-playbook deploy.yml took 45 minutes.

Because the playbook lacked proper idempotency guards, if a task failed at minute 42 due to a typo in an NSX-T REST API payload, you couldn’t simply fix the typo and rerun the playbook. You had to tear down the entire lab environment and start the 45-minute deployment from scratch.

---

The Mess: The 3-Day Variable Collision Hunt

The breaking point came during a sprint deadline for a major financial client review.

The automation pipeline kept failing at the very last step: BGP neighbor adjacency verification on the NSX-T Tier-0 Gateway.

A senior engineer spent three full days debugging the BGP configuration on the virtual router. The BGP configuration syntax was textbook perfect, but the router refused to peer with the physical switch.

# Debugging the BGP failure in the Ansible output log
TASK [nsxt_edge : Configure BGP Neighbor] **************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "BGP Peer ASN 65000 mismatch with Local ASN 65100"}

I decided to trace the variable execution graph.

On line 42 of deploy.yml, asn was defined as 65000 for the physical core router. On line 2,891—inside an unrelated task deploying a test Linux container—another engineer had declared asn: 65100 as a local loop variable.

Because Ansible playbooks share a global variable namespace by default, line 2,891 silently overwritten the BGP ASN variable set on line 42.

We had spent 72 hours chasing a ghost caused by global scope variable pollution.

---

The Solution: Modular Galaxy Roles & Isolated State Files

We halted all feature development for one week and refactored the entire automation suite into modular Ansible Galaxy roles and collections.

# # requirements.yml for Modular SDDC Automation
collections:
  - name: community.vmware
    version: 3.2.0
  - name: community.general
    version: 5.0.0
  - name: ansible.posix
    version: 1.4.0

We completely decoupled environment configuration variables from execution logic:

1. answerfile.yml: Contains strictly environment-specific variables (IP pools, VLAN IDs, ASNs, credentials). No tasks allowed.
2. Modular Roles: Split execution logic into independent, single-responsibility roles:
   • roles/esxi_prep: Configures physical host networking and storage mounts.
   • roles/vcenter_deploy: Handles vSphere Appliance deployment and cluster creation.
   • roles/nsxt_fabric: Provisions NSX Manager, Transport Nodes, and Edge Clusters.
   • roles/bgp_steering: Establishes BGP adjacencies and route maps.

# Clean, modular site.yml orchestration playbook
- name: Deploy Complete SDDC Infrastructure
  hosts: localhost
  vars_files:
    - answerfile.yml
  roles:
    - role: esxi_prep
      tags: ['esxi']
    - role: vcenter_deploy
      tags: ['vcenter']
    - role: nsxt_fabric
      tags: ['nsxt']
    - role: bgp_steering
      tags: ['routing']

By adding execution tags and strict failed_when idempotency checks, engineers could target and rerun specific sub-system roles in under 2 minutes without re-executing the entire 45-minute pipeline.

---

The Impact

• Deployment Velocity: Cut lab setup time from 45 minutes (monolithic) to 18 minutes (modular with tagged execution).
• Zero Variable Pollution: Modular role scopes eliminated variable collision bugs across the entire codebase.
• Code Reusability: The nsxt_fabric role was reused across four separate client projects without modifying a single line of task logic.

Key Takeaway

Decouple Infrastructure State Parameters from Execution Tasks.

Never let IaC playbooks grow into monolithic mega-files. Modularize automation code into clean Ansible Galaxy roles, isolate environment variables inside dedicated state files (answerfile.yml), and enforce strict role-scoped variable namespaces to prevent silent state corruption.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin