“Virtualizing high-performance Oracle RAC database clusters onto hypervisors can degrade disk I/O and trigger licensing penalties. Connecting physical bare-metal database servers directly into cloud VXLAN overlays using Hardware-VTEP bridges delivers sub-millisecond line-rate performance.”

In June 2019, during my tenure as a VMware NSX Consultant at IBM Cloud, we led the hybrid cloud migration architecture for an enterprise financial client.

The client was migrating their core e-commerce and banking application tier into an IBM Cloud (SoftLayer) VMware SDDC environment.

However, their primary database engine—a high-performance 4-node Oracle RAC (Real Application Clusters) database—had to remain on dedicated physical bare-metal hardware due to extreme disk I/O demands and strict per-core Oracle processor licensing constraints.

The challenge: How do you connect cloud-native virtual machines running inside VMware software-defined overlay subnets directly to physical bare-metal database servers without introducing latency?

---

The Software Router Bottleneck

The client’s initial internal infrastructure team proposed a traditional networking approach: route all cloud-to-database traffic through a software router VM or virtual IPsec gateway over an IP tunnel.

We built a staging test to benchmark the software routing approach.

The performance metrics were unacceptable for production banking:

1. Transaction Latency Spikes: Encapsulating high-frequency Oracle SQL query traffic through a software router VM added 12ms of latency per transaction. Oracle RAC node heartbeats missed deadlines, triggering node eviction reboots!
2. MTU Packet Fragmentation Trap: When cloud VMs sent standard 1500-byte TCP packets, the software router encapsulated them into VXLAN packets (adding 50 bytes of UDP header).

Because the physical Direct Link circuit was capped at standard MTU 1500, the 1550-byte VXLAN packets were fragmented at the network layer:

# Diagnostic Ping across IBM Direct Link showing Packet Fragmentation
ping -s 1472 -M do 192.168.200.10
# Output: Packet needs to be fragmented but DF set.
# Result: Database backup throughput collapsed from 10 Gbps down to 120 Mbps!

Packet fragmentation thrashed physical switch CPUs and dropped database query throughput by 98%.

---

The Solution: Arista Hardware-VTEP & IBM Direct Link 10Gbps

We scrapped the software routing approach and engineered a Hardware-VTEP (VXLAN Tunnel Endpoint) Bridge topology using physical Arista 7050SX Top-of-Rack switches and IBM Direct Link 10Gbps dedicated circuits.

# Hardware-VTEP Hybrid Architecture

1. **IBM Direct Link 10Gbps:** Dedicated, unthrottled private fiber connection between IBM Cloud SDDC and the client's physical colocation cage.
2. **Arista Hardware-VTEP:** Physical Arista 7050SX switch ASIC terminates VXLAN VNI `5001` in hardware memory at line rate.
3. **Seamless Layer-2 Extension:** Physical Oracle RAC servers connected to Arista ports (`VLAN 100`) communicate directly with cloud VMs on VNI `5001` on the same subnet with zero routing hops!

! # Arista ToR 7050SX OVSDB HW-VTEP Configuration for IBM Cloud
interface Vxlan1
   vxlan source-interface Loopback0
   vxlan udp-port 4789
   vxlan vlan 100 vni 5001
   vxlan flood vtep 192.168.250.10  # NSX Controller VTEP Address
!
interface Ethernet1/1
   description "Direct Link to Physical Oracle RAC Node 01"
   switchport mode access
   switchport access vlan 100
   mtu 9000 # Enforce Jumbo Frames across physical fabric

Eliminating Fragmentation with MTU 9000 Jumbo Frames

To permanently prevent MTU fragmentation, we configured MTU 9000 (Jumbo Frames) across all physical Arista switch ports, vSphere Distributed Switches (vDS), and IBM Direct Link cross-connect circuits.

This provided an extra 7,500 bytes of headroom, easily accommodating VXLAN outer UDP headers without fragmenting packets.

---

The Impact

• Sub-Millisecond Latency: Reduced cloud-to-bare-metal database latency from 12ms (software router) to 0.4 milliseconds (hardware ASIC bridging).
• 10Gbps Line-Rate Throughput: Restored full 10Gbps line-rate database query performance across the IBM Direct Link circuit.
• Zero License Penalties: Kept Oracle RAC running on dedicated physical hardware, saving the client over $1.2 Million USD in potential cloud vCPU Oracle licensing fees.

Key Takeaway

Bridge Bare-Metal Databases into Cloud Overlays using Hardware-VTEPs.

Never route high-throughput, latency-sensitive database traffic through software router VMs. When integrating physical bare-metal databases like Oracle RAC into cloud environments, deploy Hardware-VTEP bridges (Arista/Cisco) over dedicated 10Gbps Direct Link / ExpressRoute circuits, and enforce MTU 9000 (Jumbo Frames) across all physical fabrics to prevent packet fragmentation.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin