“Developers expected Kubernetes pods to spin up in milliseconds. Network security demanded every pod have an explicit IP address and an audited firewall policy. The resulting clash stalled enterprise container adoption for six months.”

In early 2022, during my tenure as a Systems Integration Advisor at NTT Data, we were deploying an enterprise OpenShift/Kubernetes platform for a major banking client.

The application development teams were thrilled. They wanted self-service container orchestration with rapid autoscaling.

The network security team was terrified.

---

The Invisible Pod Problem

Standard Kubernetes CNI plugins (like Flannel or basic Calico overlay modes) encapsulate pod-to-pod traffic inside an overlay network running on the worker node VMs.

To the physical network and the hypervisor firewall, all traffic leaving a worker node appeared to originate from the worker node’s single IP address.

If Pod-A (a public-facing web frontend) and Pod-B (an internal payment processing service) resided on the same worker node, they communicated over a local virtual bridge (cbr0) inside the Linux kernel—completely bypassing external firewalls.

Security auditing caught it immediately: Critical Compliance Defect — Zero IP visibility and no East-West microsegmentation between container workloads.

---

The Mess: The hostNetwork Workaround That Broke Scaling

The security team proposed a brute-force fix: force all Kubernetes deployments to use hostNetwork: true and allocate static IP addresses from the datacenter subnet to every pod.

We tried it on a staging cluster to test feasibility.

It was an engineering nightmare.

# The failed workaround manifest: forcing pods onto host networking
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  template:
    spec:
      hostNetwork: true # Caused port collisions and exhausted subnet IPs!
      containers:
        - name: payment
          image: payment:v1.2

Within 48 hours:

• Subnet Exhaustion: The /24 datacenter subnet ran out of IP addresses after spinning up just 200 pods.
• Port Collisions: Two pods bound to port 8080 could no longer schedule onto the same worker node, causing Kubernetes pod scheduling to fail with NodePortConflict.
• Autoscaling Failure: Horizontal Pod Autoscaling (HPA) broke completely because new pods couldn’t claim IP reservations fast enough.

We had destroyed the main reason for using Kubernetes in the first place: rapid, elastic scaling.

---

The Solution: The NSX Container Plugin (NCP CNI)

We threw out the hostNetwork workaround and deployed the NSX Container Plugin (NCP CNI).

NCP acts as a bridge between the Kubernetes API server and the NSX-T Policy API. When a developer creates a namespace or deploys a pod, NCP automatically provisions network topology inside NSX-T:

1. Namespace Isolation: NCP provisions an explicit NSX-T Logical Segment for each Kubernetes namespace.
2. Pod IP Routing: Each pod receives a routable IP address from an NSX-T IP Pool, making every pod visible to enterprise network monitoring tools like NetFlow and vRNI.
3. Automated DFW Translation: When a developer submits a native Kubernetes NetworkPolicy manifest, NCP translates it into hypervisor-enforced NSX-T Distributed Firewall (DFW) rules in real-time.

# Kubernetes NetworkPolicy Manifest Translated by NCP into DFW Rules
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: restrict-db-access
  namespace: payment-prod
spec:
  podSelector:
    matchLabels:
      role: database
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              role: payment-api
          ports:
            - protocol: TCP
              port: 5432

NCP monitors the Kubernetes API server for this manifest and issues a REST API call to NSX-T Manager:

// # NSX-T Policy API Payload Generated Automatically by NCP
{
  "display_name": "k8s-payment-prod-restrict-db-access",
  "rules": [
    {
      "display_name": "Allow-PaymentAPI-to-DB",
      "action": "ALLOW",
      "sources": ["/infra/domains/default/groups/k8s-pod-role-payment-api"],
      "destinations": ["/infra/domains/default/groups/k8s-pod-role-database"],
      "services": ["/infra/services/PostgreSQL"]
    }
  ]
}

Because the rules are enforced at the hypervisor kernel (vSwitch) level, traffic between Pod-A and Pod-B is filtered even if both pods reside on the same physical ESXi host.

---

The Impact

• Security Compliance: Delivered 100% IP visibility and hypervisor-enforced microsegmentation for all containerized application pods.
• Developer Agility: Developers continued writing standard Kubernetes NetworkPolicy YAML without needing to understand NSX-T syntax.
• IP Efficiency: Reclaimed 80% of datacenter subnet IPs by moving pod networks onto isolated NSX-T overlay segments.

Key Takeaway

Extend Enterprise Security Policy to the Container Layer.

Do not compromise container agility by forcing static networking onto Kubernetes clusters, and do not compromise security by running unmonitored overlay CNIs. Use native integration plugins like NCP CNI to translate container policies into hypervisor-enforced firewall rules automatically.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin