Load Balancer

A single server can only handle so many simultaneous connections before its CPU, memory, or network interface becomes the bottleneck — and a single server is also a single point of failure, since anything that takes it down takes the whole service with it. A load balancer solves both problems at once: it sits in front of a group of servers, spreads incoming traffic across them, and stops sending traffic to any server that stops responding correctly.

Every load balancer is a reverse proxy; not every reverse proxy balances load

A load balancer is a specific case of a reverse proxy — a device that receives connections on the client's behalf and forwards them to a backend the client never sees directly — with the added job of choosing which backend for every new connection or request. A reverse proxy that always forwards to the same single backend is doing half the job; a load balancer adds the decision logic on top.

Where that decision gets made, and how much of the traffic the load balancer actually has to understand to make it, splits load balancers into two fundamentally different designs.

Layer 4: balancing on addresses and ports alone

A Layer 4 load balancer works at the transport layer — it sees IP addresses, ports, and TCP or UDP segments, and nothing above that. When a new connection arrives, it picks a backend using whatever algorithm it's configured with, then forwards every subsequent segment on that connection to the same backend without inspecting what's inside them at all.

Client ──TCP SYN──▶ L4 load balancer ──picks backend once──▶ Backend A
Client ◀──────────────── all further segments on this connection ────────────────▶ Backend A

Because it never reads past the transport headers, an L4 load balancer genuinely doesn't know or care whether it's carrying HTTP, a raw TCP socket, gRPC, or a PostgreSQL connection — it treats every protocol identically, as an undifferentiated stream of segments belonging to one connection. That protocol-agnosticism is the main strength: it works with anything that runs over TCP or UDP, with very little per-packet overhead, and — because it never decrypts anything — it never needs to hold a TLS certificate at all.

The cost of that same ignorance is that it can't make any decision based on content. It can't route /api/users to one backend pool and /api/orders to another, can't cache a response, and can't inspect a request for anything resembling malicious content — all of that requires reading data it deliberately never looks at.

Layer 7: balancing on the actual request

A Layer 7 load balancer terminates the client's connection itself, reads enough of the actual request to understand it — an HTTP method, a path, headers, a body — and only then opens (or reuses) a separate connection to whichever backend it decides should handle that specific request. Because it's reading full requests rather than raw segments, a single client connection can have its requests spread across different backends over time, something an L4 load balancer structurally cannot do, since an L4 device never looks deep enough to tell one request from the next on the same connection.

Client ──HTTP GET /pictures──▶ L7 LB reads the request ──▶ Backend A (pictures service)
Client ──HTTP GET /orders───▶ L7 LB reads the request ──▶ Backend B (orders service)

This is what makes content-aware routing, response caching, and request-level authentication possible at the load balancer itself — and it's also why an L7 load balancer needs the TLS certificate for whatever it's fronting, exactly like a reverse proxy terminating HTTPS: it can't read an encrypted request without first decrypting it, which means the private key has to live on the load balancer. Some teams are uncomfortable with that concentration of trust, which is a real factor in choosing between the two designs, not just a performance one.

How a backend gets picked

Both designs need an algorithm to choose among healthy backends. The common ones:

• Round robin — cycle through backends in order. Simple, and fine when every backend has roughly equal capacity and every request costs roughly the same.
• Least connections — send the next connection to whichever backend currently has the fewest active ones. This adapts automatically when some requests are much more expensive than others and round robin would otherwise pile slow requests onto an already-busy server.
• IP hash / consistent hashing — derive the backend from a hash of the client's IP (or another key), so the same client consistently lands on the same backend without the load balancer needing to store any session state at all.
• Weighted variants of any of the above, when backends genuinely have different capacity — a newly added, more powerful server can be given a higher weight so it receives a proportionally larger share.

Whichever algorithm is running, it only ever considers backends the load balancer currently believes are healthy — which raises the obvious next question: how does it know?

Health checks: the difference between "up" and "actually working"

A load balancer periodically probes each backend — an L4 balancer with a plain TCP connection attempt, an L7 balancer typically with an actual HTTP request to a dedicated endpoint like /health — and stops routing traffic to any backend that fails enough consecutive checks. A backend that accepts TCP connections but whose application has deadlocked or lost its database connection would look perfectly healthy to a TCP-only check while serving nothing but errors, which is exactly why most production HTTP load balancers check for an actual 200 OK from an application-level endpoint rather than just confirming the port is open.

Practical scenario: a deploy that looks fine until the third request

A team rolls out a new backend behind an existing L7 load balancer using round robin, no session persistence configured. Login works. The dashboard loads. Then, intermittently, users get logged out mid-session for no visible reason, and the team can't reproduce it reliably in testing.

The application stores session data in server memory rather than a shared store — each backend only knows about the sessions it personally created. Round robin doesn't care about that; it spreads each new request across backends indifferent to which one issued the session cookie. A user's first request creates a session on Backend A; their second request, round-robined to Backend B, arrives at a server that's never heard of that session and treats them as logged out. It "looks fine until the third request" because with only two backends, there's roughly even odds any given follow-up request happens to land back on the right one — which is exactly what makes this bug so hard to reproduce deliberately and so persistent in production.

Two fixes actually address the cause, and they're not the same fix wearing different names. Sticky sessions (session affinity) configure the load balancer to route a given client's requests to the same backend consistently, usually via a cookie the load balancer itself sets — this works, but it re-introduces a dependency on one specific backend staying up for that client. The more robust fix is making the backends themselves stateless, storing session data in a shared store like Redis that every backend can reach, so any backend can serve any request correctly regardless of which one handled the last one. Sticky sessions patch the symptom; shared session storage removes the reason it existed.

Practice exercises

1. Explain why a Layer 4 load balancer cannot route requests to different backends based on URL path, while a Layer 7 load balancer can — tie the answer to exactly what data each one is reading.
2. A backend's process is alive and its TCP port is open, but its database connection pool is exhausted and every request it receives times out. Explain why a TCP-only health check would fail to catch this, and what kind of check would.
3. A load balancer using least-connections sends a disproportionate number of requests to one backend that turns out to be your newest, most powerful server. Is this a bug? Explain what's actually happening.

A load balancer decides which backend handles a request, and it does that job invisibly to the client — the client only ever sees the load balancer's own address, never the backend it actually reached. That's one specific case of a much older and more general idea: a server that makes a decision or a request on someone else's behalf without the other side needing to know the details. The next two articles cover that idea in both of its directions, starting with the one where it's the client, not the server, that deliberately routes through a middleman.

Sources

• NGINX Documentation, HTTP Load Balancing
• Cloudflare Learning Center, What is load balancing? | How load balancers work