“Running a 40Gbps Oracle RAC database through a software L2 Bridge VM will melt your virtual CPUs and trigger cluster split-brain failures. When virtual overlays need to talk to bare-metal hardware at line rate, you need Hardware VTEPs.”

In early 2020, during my tenure as a VMware NSX Consultant at IBM, we were architecting a private cloud SDDC for a major banking client.

The application tier was 100% virtualized on VMware vSphere with NSX-V overlay networking. But the financial core database—a massive physical Oracle RAC cluster—remained on bare-metal servers connected directly to physical Arista 7050 switches.

The database team had a non-negotiable requirement: the virtual application servers and physical Oracle RAC nodes had to reside on the exact same Layer-2 broadcast domain (10.100.50.0/24).

No routing boundaries. No NAT. Same subnet.

---

The Limits of Software Encapsulation

Virtual machines inside an NSX overlay communicate using VXLAN encapsulation. The hypervisor’s virtual switch (vDS) encapsulates Ethernet frames into UDP packets before sending them across the physical network.

Physical servers, however, have standard network cards (NICs) that have no idea what VXLAN is. They expect plain IEEE 802.1Q Ethernet frames.

To bridge virtual and physical hosts on the same subnet, you need an endpoint that can strip the VXLAN header from incoming packets and tag outbound frames with the correct VLAN ID before forwarding them to the physical server.

---

The Mess: The Software Bridge That Melted Under Load

Our first implementation relied on the standard NSX-V Software L2 Bridge instance.

The L2 Bridge is a specialized virtual machine running on an ESXi host that bridges a virtual VXLAN wire to a physical VLAN port.

During staging tests, it seemed fine. But when we initiated the full 40Gbps database stress-test pass, the software bridge collapsed.

# Monitoring the NSX Software L2 Bridge VM during stress testing
esxtop -> vcpu stats
# Output: CPU %UTIL: 100.00% | Packet Drops: 35.4% | RX Drop Rate: 120,000 pps

The virtual CPU on the bridge VM hit 100% utilization. Processing 40Gbps of high-throughput Oracle inter-node cache fusion traffic completely saturated the VM’s vCPUs.

The bridge began dropping 35% of incoming packets.

The dropping packets caused Oracle RAC nodes to miss heartbeat pings, triggering a split-brain quorum panic. The database cluster locked up and fence-rebooted two of the physical nodes.

Software bridging could not scale to physical database speeds.

---

The Solution: OVSDB Peering with Arista Hardware VTEPs

We eliminated the software bridge VM entirely and integrated the physical Arista 7050 switches directly into the NSX control plane as Hardware VTEPs (HW-VTEP).

Instead of an ESXi vCPU doing the packet encapsulation, the physical Arista switch hardware ASICs performed VXLAN encapsulation and decapsulation at native 40Gbps line rate.

The integration was powered by the OVSDB (Open vSwitch Database) protocol.

# Arista EOS OVSDB & VXLAN Hardware VTEP Configuration
cvx
  no shutdown
  server 10.100.1.10 # Points to NSX Controller Cluster IP
!
interface Vxlan1
  vxlan source-interface Loopback1
  vxlan udp-port 4789
  vxlan vlan 50 vni 10050

The OVSDB Control Plane Exchange

1. Control Plane Registration: The Arista switch establishes a TLS-encrypted OVSDB connection to the NSX Controller cluster (10.100.1.10).
2. MAC Binding Exchange: When a physical Oracle node connected to Arista port Ethernet1/1 sends an ARP frame, the Arista switch learns the MAC address and pushes it over OVSDB to the NSX Controllers.
3. Hardware-Line-Rate Encapsulation: When a VM sends a packet to the physical Oracle node, the ESXi host encapsulates the frame into VXLAN and sends it to the Arista switch’s loopback IP. The Arista ASIC strips the VXLAN header and delivers raw 802.1Q frames to the physical server in sub-microseconds.

# Verifying HW-VTEP OVSDB Binding Status on Arista Switch
show cvx connections
# Output: Controller: 10.100.1.10 | Status: ESTABLISHED | Active VTEP Bindings: 12

---

The Impact

• Zero CPU Overhead: Offloaded 100% of packet encapsulation from hypervisor vCPUs to physical switch ASICs.
• Line-Rate Throughput: Achieved full 40Gbps line-rate forwarding between virtual app servers and physical Oracle RAC nodes with sub-microsecond latency.
• Eliminated Split-Brain Drops: Zero packet drops recorded during 72-hour maximum stress-testing passes.

Key Takeaway

Use Hardware VTEPs for High-Throughput Bare-Metal Integration.

Do not rely on software bridge VMs for high-throughput database workloads. When physical bare-metal servers must join a software-defined overlay network on the same Layer-2 subnet, peer your physical switches with the SDN controller using OVSDB to let hardware ASICs handle packet encapsulation at line rate.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin