---
topic: cloud-infrastructure
author: Crashtech Editorial
published: Nov 15, 2026 · read: 7 min
---

Client-Side Load Balancing on Kubernetes: Why gRPC Traffic Skews and How Databricks Routes Per Request

How Databricks eliminated gRPC traffic skew in Kubernetes: custom xDS control plane, client-side P2C routing, zone-affinity, and slow-start ramp-up.

– –

Kubernetes default networking primitives (ClusterIP services, CoreDNS, and kube-proxy iptables or IPVS modes) form the foundation of container routing. For traditional stateless HTTP/1.1 microservices that open and close TCP connections frequently, Layer 4 connection-level packet distribution provides adequate load dispersal.

However, modern high-throughput architectures rely heavily on gRPC running over persistent HTTP/2 multiplexed connections. Under persistent streaming protocols, Kubernetes default connection-level routing fundamentally breaks: a single TCP stream remains pinned to one backend pod for hours or days.

At Databricks scale, this connection pinning created severe operational pathologies: heavy tail latencies, pod utilization imbalances where 20% of pods handled 80% of traffic, and costly resource over-provisioning.

To overcome these architectural constraints, Databricks shifted routing authority from infrastructure proxies into client runtimes. By pairing a custom Kubernetes Endpoint Discovery Service (EDS) control plane with client-side Power of Two Choices (P2C) and zone-affinity algorithms, Databricks eliminated traffic skew and cut cluster pod requirements by 20 percent.

+-----------------------------------------------------------------------------------+
|               TRADITIONAL K8S L4 (SKEWED) VS DATBRICKS L7 CLIENT LB               |
+-----------------------------------------------------------------------------------+

[ Traditional Kube-Proxy L4 Routing ]
Client App ----(Persistent TCP)----> kube-proxy (iptables) ----> Pod 1 (100% Load)
                                                                 Pod 2 (0% Idle)
                                                                 Pod 3 (0% Idle)

[ Databricks Client-Side L7 Routing ]
                     +----------------------------+
                     | Databricks Control Plane   |
                     | (Watches EndpointSlices)   |
                     +----------------------------+
                                   | xDS / EDS Streaming Updates
                                   v
Client RPC Library (Maintains In-Memory Endpoint Pool & P2C)
   |-- RPC Request 1 -----------------------------------------> Pod 1 (33% Load)
   |-- RPC Request 2 -----------------------------------------> Pod 2 (33% Load)
   \-- RPC Request 3 -----------------------------------------> Pod 3 (33% Load)

1. The Anatomy of gRPC Traffic Skew in Kubernetes

In default Kubernetes networking, a service name (such as catalog.prod.svc.cluster.local) resolves via CoreDNS to a single virtual ClusterIP. When an application initiates a TCP connection to this virtual IP:

  1. Linux kernel iptables or IPVS rules configured by kube-proxy intercept the initial TCP SYN packet.
  2. A destination network address translation (DNAT) selects one target pod IP from the service endpoint list.
  3. Every subsequent IP packet belonging to that TCP connection follows the established kernel NAT connection-tracking table (conntrack).
TCP SYN Packet ---> iptables PREROUTING ---> DNAT 10.244.1.15:50051 (Conntrack Entry Created)
Subsequent Data ---> Bypasses selection ---> Forwards to 10.244.1.15:50051 directly

Why HTTP/2 Breaks Layer 4 Assumptions

HTTP/1.1 clients typically configure connection pools that open fresh TCP sockets or cycle connections every few requests. In contrast, gRPC utilizes HTTP/2 connection multiplexing:

  • Single Long-Lived Socket: A client establishes a single persistent TCP connection to a server and multiplexes hundreds of concurrent bidirectional RPC streams over it.
  • Static Conntrack Pinning: Because the TCP connection never closes, kube-proxy never re-evaluates backend selection.
  • Horizontal Pod Autoscaling Failure: When a service autoscales from 10 to 30 pods to handle a traffic surge, existing clients continue pumping all RPCs over their already-established TCP connections to the original 10 pods. The 20 newly spawned pods sit completely idle while original pods suffer OOM kills or CPU throttling.
+-----------------------------------------------------------------------------------+
|             THE SCALING PARADOX: NEW PODS REMAIN STARVED UNDER HTTP/2             |
+-----------------------------------------------------------------------------------+

Established Clients
[ Client A ] ===========(1 Long-lived TCP)===========> [ Pod 1: 95% CPU (Hot) ]
[ Client B ] ===========(1 Long-lived TCP)===========> [ Pod 2: 98% CPU (Hot) ]

Autoscaler deploys new capacity:
                                                        [ Pod 3:  2% CPU (Idle) ]
                                                        [ Pod 4:  1% CPU (Idle) ]
                                                        [ Pod 5:  1% CPU (Idle) ]
(Clients never disconnect to discover new pods)

2. Moving Routing Authority to the Client

Engineering teams traditionally address this problem by inserting Layer 7 reverse proxies (such as NGINX or Envoy sidecars) between services. While a sidecar mesh inspects HTTP/2 frames and distributes individual RPCs, it introduces two distinct infrastructure taxes:

  1. Network Double-Hopping: Every RPC traverses two additional localhost network hops and kernel socket buffers (Client Pod -> Client Sidecar -> Server Sidecar -> Server Pod).
  2. Resource Bloat: Deploying sidecar containers alongside tens of thousands of microservice pods consumes massive CPU and memory overhead simply maintaining duplicate connection pools and control plane states.

Databricks chose a client-library architecture. Instead of inserting external proxy processes, they embedded Layer 7 routing logic directly into their shared Scala and Java RPC client SDKs.

+-----------------------------------------------------------------------------------+
|            DATABRICKS INTELLIGENT LOAD BALANCING ARCHITECTURAL FLOW               |
+-----------------------------------------------------------------------------------+

                    +-----------------------------+
                    |    Kubernetes API Server    |
                    | (EndpointSlices Resources)  |
                    +-----------------------------+
                                   | Informer Watch
                                   v
                    +-----------------------------+
                    | Databricks EDS Control Plane|
                    |  - Zone Topology Enrichment |
                    |  - Health State Tracking    |
                    |  - xDS Protocol Streamer    |
                    +-----------------------------+
                                   |
                  +----------------+----------------+
                  | gRPC xDS Stream                 | gRPC xDS Stream
                  v                                 v
   +------------------------------+   +------------------------------+
   | Databricks Client RPC Lib    |   | Ingress Gateway (Envoy)      |
   |  - Active Endpoint Pool      |   |  - External Ingress Routing  |
   |  - P2C Selection Engine      |   |  - Unified Endpoint Truth    |
   |  - Zone-Affinity Filters     |   +------------------------------+
   +------------------------------+
                  |
        +---------+---------+
        |                   |
        v                   v
  [ Backend Pod A ]   [ Backend Pod B ]
   (Zone: us-west-2a)  (Zone: us-west-2b)

3. The Custom xDS Endpoint Discovery Service (EDS)

To make intelligent routing decisions, the client must possess real-time visibility into the physical topology of backend pods. Relying on periodic DNS SRV queries is unfeasible: DNS queries induce network overhead, lack sub-second propagation speed, and cannot communicate rich metadata like availability zone or warm-up state.

Databricks built a custom, lightweight control plane implementing the Envoy Discovery Service (EDS) specification (part of the universal xDS protocol).

Control Plane Workflow:

  1. EndpointSlice Watching: The control plane establishes persistent WebSocket/gRPC Informer watches against the Kubernetes API server, monitoring discovery.k8s.io/v1 EndpointSlice objects.
  2. Metadata Enrichment: The control plane enriches raw pod IP addresses with critical infrastructure metadata:
    • AWS / Azure Availability Zone (topology.kubernetes.io/zone)
    • Pod Readiness status (conditions.ready == true)
    • Shard identifiers and service-specific traffic weights
  3. Reactive xDS Streaming: When pods spin up or terminate, the control plane pushes delta updates over an active HTTP/2 bidirectional stream directly to connected clients. Zero polling is required.
// Simplified JSON representation of xDS EDS ClusterLoadAssignment
{
  "cluster_name": "service-model-serving",
  "endpoints": [
    {
      "locality": { "zone": "us-west-2a" },
      "lb_endpoints": [
        {
          "endpoint": { "address": { "socket_address": { "address": "10.244.3.42", "port_value": 9000 } } },
          "health_status": "HEALTHY",
          "load_balancing_weight": 100
        }
      ]
    }
  ]
}

4. Advanced Request-Level Routing Algorithms

With real-time endpoint topology loaded in memory, Databricks client libraries execute dynamic, request-level load balancing algorithms for every outgoing RPC.

1. Power of Two Choices (P2C) with Least Outstanding Requests

Standard round-robin routing treats all backend pods as identical compute units. If Pod 3 encounters a heavy analytical query or a minor garbage collection pause, round robin continues blindly feeding it requests, triggering latency spikes.

Databricks implemented Power of Two Choices (P2C):

Algorithm: Power of Two Choices (P2C)

Input: Set of healthy backend endpoints E = {e_1, e_2, ..., e_n}
Output: Target endpoint e_target

1. Select two distinct endpoints uniformly at random: e_a, e_b in E
2. Query in-flight request counters:
   load_a = active_requests(e_a)
   load_b = active_requests(e_b)
3. If load_a <= load_b:
   Return e_a
   Else:
   Return e_b

By comparing two random candidates rather than scanning all $N$ hosts, P2C executes in constant $O(1)$ CPU time while providing near-optimal load balancing guarantees, preventing herd effects.

// Architectural abstraction of P2C Client Selection in Scala
case class Endpoint(address: String, activeRequests: AtomicInteger, zone: String)

class P2CLoadBalancer(endpoints: Vector[Endpoint]) {
  private val rng = new Random()

  def selectEndpoint(): Endpoint = {
    val size = endpoints.length
    if (size == 0) throw new NoAvailableEndpointsException()
    if (size == 1) return endpoints(0)

    val idx1 = rng.nextInt(size)
    var idx2 = rng.nextInt(size - 1)
    if (idx2 >= idx1) idx2 += 1

    val ep1 = endpoints(idx1)
    val ep2 = endpoints(idx2)

    if (ep1.activeRequests.get() <= ep2.activeRequests.get()) ep1 else ep2
  }
}

2. Zone-Affinity Routing with Controlled Spillover

Cross-AZ network traffic in public cloud environments incurs financial transfer fees and adds 1 to 2 milliseconds of physical fiber propagation latency.

Databricks clients enforce Zone-Affinity Routing:

  • Primary Preference: Clients exclusively route RPCs to backend pods located in the identical availability zone (client_zone == server_zone).
  • Dynamic Spillover: The client monitors local zone capacity. If local zone pods become saturated or fall below healthy readiness thresholds, traffic gracefully spills over to adjacent zones according to pre-allocated weight ratios.

5. Unifying Internal and External Ingress via Envoy

Rather than maintaining separate routing stacks for internal cluster traffic and external user traffic, Databricks integrated their EDS control plane directly with perimeter Envoy Ingress Gateways.

External Customer Request ---> Envoy Ingress Gateway --(Subscribes to EDS)--> Target Pod (Direct)
Internal Service Request  ---> Scala RPC Client      --(Subscribes to EDS)--> Target Pod (Direct)

Because external Envoy instances and internal Scala clients subscribe to the exact same xDS control plane:

  • Both tiers operate on identical endpoint health states and zone-topology maps.
  • Pod deployments and rollouts reflect immediately across both edge ingress and inter-service topologies without synchronization skew.

6. Hard-Won Operational Lessons

Transitioning to client-side load balancing at enterprise scale revealed nuanced edge cases that required architectural defenses:

1. The Cold-Start Shock and Slow-Start Ramp-Up

When a new backend pod becomes Ready, its active in-flight request counter is zero (active_requests = 0). Under pure P2C, the algorithm aggressively routes traffic to the newly available pod because its load appears lower than warmed peers.

However, newly initialized JVM runtimes require JIT compilation and cache warming:

  • Flooding an un-warmed pod with peak traffic instantly causes thread pool exhaustion and timeout cascades.
  • The Solution: Databricks implemented a Slow-Start Ramp-Up algorithm. When a pod first transitions to healthy, the client artificially throttles its selection weight, linearly ramping traffic from 0% to 100% over a 60-second window.

2. Failure of CPU/Memory Metrics for Routing

Databricks initially experimented with metric-based load balancing, attempting to feed host CPU and memory utilization into routing calculations:

  • Lagging Indicator Trap: Metric ingestion pipelines (cAdvisor, Prometheus, metrics-server) report utilization with 5 to 15 seconds of latency.
  • By the time a client noticed a pod’s CPU spiked to 90%, the condition had either already passed or compounded into severe queue backlogs.
  • The Shift: Databricks replaced telemetry-driven routing with immediate Layer 7 signals: in-flight request counts, connection error rates, and HTTP/2 stream reset codes.

Architectural Comparison: K8s Default vs Databricks Client LB

Architectural DimensionKubernetes Default (kube-proxy)Databricks Intelligent Client LB
Inspection LayerLayer 4 (TCP / IP packets)Layer 7 (Individual RPC frames)
Protocol BehaviorBalances once per TCP handshakeBalances every individual RPC request
HTTP/2 & gRPC FitSevere traffic skew on persistent socketsPerfectly balanced across all pods
Topology AwarenessPrimitive / Static iptablesDynamic Zone-Affinity with spillover
Routing AlgorithmRandom or Round-RobinPower of Two Choices (P2C) + Slow-Start
Resource EfficiencyRequires 20-30% over-provisioning20% reduction in fleet pod count

Summary Takeaways

Databricks architectural evolution highlights critical lessons for high-throughput cloud systems:

  • L4 proxies fail long-lived protocols: Persistent HTTP/2 and gRPC streams defeat connection-level packet routers like kube-proxy, demanding Layer 7 request granularity.
  • Client-side routing eliminates proxy tax: Embedding routing inside shared client SDKs delivers fine-grained Layer 7 traffic management without the CPU and latency penalties of sidecar proxies.
  • Standardize on xDS: Adopting the open xDS protocol allows custom control planes to seamlessly control both internal microservice clients and edge Envoy proxies from a unified source of truth.
  • Defend against cold starts: Any dynamic load balancer that favors lightly loaded endpoints must implement slow-start ramp-up to protect newly deployed instances from immediate traffic drowning.
Advertisement

Frequently asked questions

Why does Kubernetes kube-proxy cause traffic skew with gRPC workloads?

gRPC operates on HTTP/2, multiplexing hundreds of RPC calls over long-lived, persistent TCP connections. Kubernetes kube-proxy load balances at Layer 4 (the initial TCP handshake). Once a TCP stream connects to a backend pod, all subsequent RPCs pin to that pod indefinitely, causing massive load imbalance.

How does client-side load balancing solve the gRPC connection pinning problem?

Client-side load balancing moves routing logic directly into the application RPC client. The client maintains an active list of healthy backend pod IPs and independently balances each individual RPC request across pods, bypassing kube-proxy connection-level pinning entirely.

What is the role of Envoy Discovery Service (EDS) and xDS in Databricks' architecture?

Databricks built a custom control plane watching Kubernetes EndpointSlices. It translates dynamic pod IP additions, removals, and zone metadata into standard xDS/EDS gRPC streams, pushing instantaneous endpoint updates directly to internal client libraries and ingress Envoy proxies.

What is the Power of Two Choices (P2C) algorithm and why is it preferred over round robin?

P2C picks two random healthy candidate endpoints on every request and dispatches the RPC to the endpoint with fewer outstanding active requests. This simple $O(1)$ decision prevents herd behavior, avoids thundering herds, and achieves near-optimal load distribution.

Sources & further reading

/* Comments */