Traffic Engineering at the Boundary: Load Balancers, Gateways and Reverse Proxies
How Layer 4 and Layer 7 balancers, edge API gateways, and reverse proxies coordinate to shape, authenticate, and distribute web-scale traffic.
On this page

When engineers discuss “load balancing”, they often conflate three distinct infrastructure tiers: the transport-level packet router (L4), the application-level content router (L7), and the business-aware API gateway. Each tier operates at a different layer of the OSI stack, incurring distinct latency, CPU, and memory trade-offs.
The Multi-Tier Ingress Funnel
High-traffic distributed applications organize edge traffic into a hierarchical funnel:
[Global Internet Ingress]
|
v
+-----------------------------------------------------------+
| Tier 1: Layer 4 Load Balancer (AWS NLB, Maglev, IPVS) |
| - Operates at Transport Layer (TCP / UDP) |
| - Fast packet forwarding via Direct Server Return (DSR) |
| - Zero payload inspection |
+-----------------------------------------------------------+
|
v
+-----------------------------------------------------------+
| Tier 2: API Gateway / Layer 7 Router (Envoy, Kong, Traefik)|
| - Terminates TLS 1.3 / HTTP/2 / HTTP/3 |
| - Enforces JWT validation & rate limiting |
| - Routes dynamically by path, host, or HTTP header |
+-----------------------------------------------------------+
|
+-----+-----+
| |
v v
+---------+ +---------+
| Reverse | | Reverse | Tier 3: Local Pod Proxies (NGINX / Envoy)
| Proxy A | | Proxy B | - Connects to local Unix sockets / apps
+----+----+ +----+----+
| |
v v
[App Node 1] [App Node 2]
Comparing the Three Core Boundary Tiers
| Dimension | Layer 4 Load Balancer | Reverse Proxy (e.g., NGINX) | API Gateway |
|---|---|---|---|
| OSI Layer | Layer 4 (Transport: TCP/UDP) | Layer 7 (Application: HTTP) | Layer 7 + Application Logic |
| Routing Granularity | IP address and port only | URL path, Host header, HTTP verbs | User claims, scopes, dynamic routing |
| Payload Visibility | None: blind to HTTP body and headers | Full HTTP inspection and header rewrite | Full payload parsing & JSON transformation |
| Throughput & Speed | Extreme (millions of PPS, sub-ms) | High (tens of thousands QPS) | Moderate (governed by CPU policies) |
| Core Responsibilities | TCP connection multiplexing, DSR | Static file serving, TLS offloading | Authentication, billing, quota throttling |
Traffic Distribution Algorithms in Production
Selecting the right balancing algorithm depends strictly on the workload profile:
1. Round Robin & Weighted Round Robin
Dispatches requests sequentially across instances.
- Best for: Stateless services with uniform request processing times (e.g., serving static asset responses).
- Failure Mode: Under requests with variable compute times (e.g., quick health checks mixed with heavy SQL report generation), round robin leads to queue imbalance on slow workers.
2. Least Connections & Weighted Least Connections
Routes the incoming connection to the server currently maintaining the fewest active open sockets.
- Best for: Long-lived HTTP/2 streams, WebSocket connections, or database query workers where request execution durations vary significantly.
3. Consistent Hashing (IP / Token Affinity)
Hashes a client attribute (IP address, session cookie, user ID) onto a consistent ring with virtual nodes.
- Best for: State-dependent caches (such as distributed caching tiers or real-time collaborative workspaces) where maximizing local cache hit ratios is paramount.
Edge Resiliency: Guarding Backends from Collapse
Do
- Terminate TLS early at the L7 gateway to offload cryptographic CPU overhead from backend application containers.
- Enforce circuit breaking and adaptive concurrency limits to drop requests immediately when downstream queues saturate.
- Use Direct Server Return (DSR) on Layer 4 balancers to let return packets bypass the balancer, maximizing egress throughput.
Don't
- Do not inspect deep HTTP request bodies at the Layer 4 tier; preserve protocol blindness for maximum throughput.
- Avoid deploying sticky sessions based on client IP addresses; clients behind corporate NAT gateways will cluster onto a single backend server.
- Never let API gateways perform slow synchronous database calls during the ingress authorization path.
By placing protocol-blind Layer 4 balancers in front of intelligent Layer 7 API gateways and reverse proxies, organizations construct boundary architectures that absorb volumetric attacks while delivering deterministic latency to millions of concurrent users.
Frequently asked questions
What is the architectural difference between an API Gateway and a Reverse Proxy?
A reverse proxy (like NGINX) terminates TLS and forwards requests directly to backends. An API gateway extends this with application-layer intelligence: validating JWT tokens, managing rate limit quotas, composing microservices, and transforming protocols.
When should an architecture use Layer 4 instead of Layer 7 load balancing?
Use Layer 4 (TCP/UDP) when raw throughput and sub-millisecond connection handling are critical, or when handling non-HTTP protocols (e.g., database wire protocols, gaming UDP streams). Use Layer 7 when routing requires reading HTTP paths, headers, or cookies.
How do edge ingress layers prevent backend cascading thundering herds?
Through connection pooling, request coalescing, circuit breaking, and load shedding. When backends experience degradation, the gateway sheds lower-priority background traffic while answering cached queries locally.
What algorithm is best for sticky sessions without breaking even distribution?
Consistent hashing on client session tokens with virtual nodes. This maintains session affinity while distributing traffic evenly and ensuring minimal key reshuffling when backend nodes scale.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.