Zero-Touch Pod Decommissioning: Automated Teardown via Ansible & NSX-T API

“Automating infrastructure deployment is only half the battle — true cloud agility requires zero-touch automated decommissioning to prevent resource sprawl.”

The Setup

In October 2021 at NTT Data, our private cloud automation platform managed short-lived staging and QA pods for enterprise software clients. Hundreds of temporary vSphere tenant pods were provisioned weekly.

While deployment was automated via Ansible Tower, decommissioning remained a manual 15-step checklist.

The Mess

Engineers frequently delayed or forgot manual teardown steps when client testing completed:

• Abandoned Tier-1 Gateways and segment ports remained active in NSX-T Manager.
• Unused IP subnets sat locked in IPAM, triggering artificial IP space exhaustion.
• Orphaned Distributed Firewall Rules accumulated in NSX-T policy tables, slowing down policy evaluation cycles across 200+ host clusters.

[WARNING] 2021-10-12 11:45:00 - NSX-T Audit Compliance Scanner
Orphaned Resources Detected:
- 42 Unattached Tier-1 Gateways (No active segment ports)
- 120 Inactive Segment Subnets holding 7,680 IP addresses
- 310 Stale Distributed Firewall Rules referencing deleted AD groups

Manual cleanup tickets took 5 days per pod, wasting thousands of dollars in un-reclaimed cloud capacity.

The Solution

I built an automated Zero-Touch Pod Teardown Workflow in Ansible Tower that executed reverse dependency unbinding and resource reclamation:

1. Cascade Dependency Deletion: Programmed Ansible tasks to delete Distributed Firewall rules first, followed by Segment interfaces, Tier-1 Gateways, and IPAM subnet releases.
2. Safety Dry-Run Mode: Added an automated check verifying zero active virtual machine NICs were attached to segment ports prior to deletion.
3. Idempotent REST API Calls: Executed DELETE requests to the NSX-T Policy API with status code validation.

# roles/nsxt_teardown/tasks/main.yml - Automated Pod Teardown
- name: Verify zero active VMs on segment ports
  uri:
    url: 'https://{{ nsxt_manager }}/policy/api/v1/infra/tier-1s/{{ tenant_id }}/segments/{{ item.name }}/realized-state'
    method: GET
    url_username: '{{ nsxt_user }}'
    url_password: '{{ nsxt_password }}'
  register: realized_state
  failed_when: realized_state.json.state.attachment_count > 0

- name: Delete Tier-1 Gateway via NSX-T Policy API
  uri:
    url: 'https://{{ nsxt_manager }}/policy/api/v1/infra/tier-1s/{{ tenant_id }}'
    method: DELETE
    url_username: '{{ nsxt_user }}'
    url_password: '{{ nsxt_password }}'
    status_code: [200, 204]

Key Takeaway

Always pair deployment automation with automated zero-touch decommissioning pipelines. Automated resource reclamation prevents IP address exhaustion and keeps SDN policy tables clean.

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin