Skip to content

Rate Limiting and API Gateway

The reverse proxy article already showed one specific way rate limiting breaks — reading the wrong source IP behind a proxy — without asking the more basic question first: how does a rate limiter actually decide a client has sent "too many" requests, and where does that decision get made? Zero Trust, the previous article in this module, answered whether a request is allowed to reach a resource at all. This article answers a narrower, equally important question: given that a request is allowed, how does a service stop it — and thousands like it — from arriving faster than the service can safely absorb, whether that flood is an honest traffic spike or a deliberate attack?

What happens with no limit at all

A backend with no rate limiting treats every incoming request identically, regardless of how many that same client already sent in the last second. That's fine right up until it isn't: a client with a bug in its retry logic, a scraper pulling every page as fast as the network allows, or an attacker deliberately hammering a login endpoint to brute-force passwords all look the same to an unprotected server — an unbounded stream of otherwise-valid requests. The server keeps accepting them until it runs out of some finite resource: database connections, worker threads, memory — at which point it stops responding to everyone, including the legitimate users the flood was never aimed at.

Rate limiting is the deliberate decision to reject some requests on purpose, so the ones that get through keep succeeding. It trades a guaranteed, controlled failure for a subset of requests against an uncontrolled failure for all of them.

Two ways to count "too many"

Every rate limiter needs an algorithm for tracking how many requests a client has made recently and deciding when that count crosses a limit. Two classic approaches show up constantly in real infrastructure, and they behave differently enough that picking the wrong one for a given workload causes real problems:

  • Token bucket. Each client has a bucket that holds up to some maximum number of tokens, refilled at a steady rate — one token every 100ms, say. Every request consumes one token; a request that arrives when the bucket is empty gets rejected. Because the bucket can fill up during quiet periods, a client that's been idle can burst a batch of requests all at once, as long as it doesn't exceed the bucket's maximum size — token bucket permits bursts, up to a cap.
  • Leaky bucket. Requests go into a queue (the "bucket"), and they're processed out of it at a fixed, steady rate — the analogy is water poured in at the top and leaking out the bottom at a constant pace regardless of how fast it's poured in. If requests arrive faster than the leak rate and the queue fills up, new requests are rejected outright. Leaky bucket smooths traffic to a constant output rate; it doesn't allow the same kind of burst token bucket does, because the processing rate itself never speeds up.

NGINX's own limit_req module, one of the most widely deployed rate limiters in production, is a direct implementation of leaky bucket:

limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

server {
    location /api/ {
        limit_req zone=api burst=20 nodelay;
    }
}

rate=10r/s sets the steady leak rate — 10 requests per second per client, tracked by $binary_remote_addr, the client's IP in compact binary form. burst=20 allows up to 20 requests beyond that steady rate to queue rather than being rejected immediately, and nodelay tells NGINX to process queued burst requests immediately, as slots become available, rather than holding each one back to strictly enforce the 10r/s pace — trading strict smoothness for handling a realistic traffic pattern, where a browser loading a page fires off a dozen requests at once rather than spacing them out evenly.

Reading a rate-limited response correctly

A client that gets rate-limited needs to know two things: that it was limited, and how long to wait before it's worth trying again. HTTP has a specific status code for exactly this, defined in RFC 6585: 429 Too Many Requests.

curl -i https://api.example.com/search?q=test

HTTP/1.1 429 Too Many Requests
Retry-After: 30
Content-Type: application/json

{"error": "rate_limit_exceeded", "retry_after_seconds": 30}

Retry-After tells the client precisely how long to wait — either a number of seconds, as here, or an absolute HTTP date — before sending another request. A well-behaved client reads this header and backs off accordingly rather than retrying immediately, which would just extend its own rate-limited window. A poorly written client that ignores 429 entirely and retries in a tight loop makes its own situation worse, and depending on how the limiter counts rejected requests, can end up extending its own penalty indefinitely.

Many APIs also expose the current state of the limit proactively, on every response, not just the one that finally gets rejected — commonly through headers like X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset. These aren't standardized the way 429 and Retry-After are — the exact header names vary by provider — but the pattern lets a well-behaved client see it's approaching its limit and slow down voluntarily, before ever receiving an actual 429.

What an API gateway actually is

Everything described so far can run as a single limit_req block on one reverse proxy in front of one service. That stops being sufficient once an organization runs many backend services behind one public-facing address — the more common shape for anything built as microservices — and needs the same policies (rate limiting, authentication, request logging) applied consistently across all of them rather than reimplemented separately in each backend's own code.

An API gateway is a reverse proxy purpose-built for exactly that: a single entry point that sits in front of many backend services and applies cross-cutting concerns centrally, before a request ever reaches the specific service that will actually handle it.

Client → API Gateway → /users  → Users service
                      → /orders → Orders service
                      → /search → Search service

Gateway handles once, for all three: auth, rate limiting, request logging, TLS termination

This is the same routing-by-path pattern the reverse proxy article already described — /api to one backend, / to another — extended into a dedicated layer that does considerably more than route. An API gateway typically also handles:

  • Authentication and authorization, checked once at the gateway rather than reimplemented in every backend service, so a backend can trust that a request reaching it already passed identity checks.
  • Rate limiting, applied per client, per API key, or per endpoint, exactly as this article has already described — but configured and enforced in one place instead of duplicated across every service.
  • Request/response transformation, adapting an external API contract to whatever internal format the backend services actually expect, so backend teams can change their own internal APIs without breaking external clients.

The trade-off is architectural, not just operational: an API gateway becomes a single point every request passes through, which makes it an obvious place to enforce policy consistently — and also a single point that, if it goes down or is misconfigured, can take every backend service behind it offline at once, even if every one of those backends is individually healthy.

Practical scenario: a partner integration that keeps tripping the limit

A company exposes a public API with a documented limit of 100 requests per minute per API key. A partner integration reports constant 429 errors, even though their own logs show they're sending well under 100 requests per minute — nowhere close to the documented number.

curl -i https://api.example.com/v1/orders \
  -H "Authorization: Bearer <api-key>"

HTTP/1.1 429 Too Many Requests
Retry-After: 45
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1732104060

X-RateLimit-Remaining: 0 confirms the limit really is exhausted — this isn't a bug in the limiter reporting a false positive. Comparing timestamps between the partner's own request log and the gateway's rate-limit window reveals the actual mismatch: the partner is counting requests over a rolling 60-second window measured from whenever they happen to check, while the gateway resets its own counter at fixed 60-second clock boundaries. A burst of requests sent right at the end of one fixed window, followed by another burst right at the start of the next, is well under 100 requests in either individual window from the gateway's point of view — but from the partner's own rolling-window accounting, those two bursts look like they're spread evenly across a full minute and nowhere near the limit.

The mismatch isn't a bug on either side; it's two different, both-legitimate ways of defining "per minute," and it's exactly the kind of detail that has to be documented explicitly in API documentation rather than left for an integrating partner to guess. The fix here is entirely about clarity, not configuration: the API documentation needs to state precisely which windowing strategy the gateway uses — fixed window, rolling window, or the token-bucket-style burst allowance NGINX's limit_req demonstrated earlier — so a partner's own client-side throttling can be built to match it instead of a different, incompatible assumption.

Practice exercises

  1. A client bursts 15 requests in the first 200ms after being idle, against a limiter configured for rate=10r/s with burst=20. Using the leaky-bucket description above, explain whether those 15 requests are rejected, and what changes if the same client tries the same burst again immediately afterward with no idle time in between.
  2. Explain, referencing the reverse proxy article's X-Forwarded-For scenario, why an API gateway that terminates client connections and forwards to backend services needs to solve the exact same source-IP problem that a plain reverse proxy does.
  3. Design, at a high level, which of the three cross-cutting concerns this article named (authentication, rate limiting, request transformation) you would centralize at an API gateway versus leave to individual backend services, for a system where one backend genuinely needs a stricter rate limit than the rest because it triggers an expensive database query per request.

Rate limiting and an API gateway's other centralized checks assume the requests arriving are honest, if excessive — a real client sending more traffic than it should. What they don't assume, and can't fully defend against alone, is an attacker whose entire goal is to send as much traffic as physically possible, from as many sources as possible, specifically to overwhelm a target rather than merely exceed its normal usage pattern. That's the last piece this module leaves to cover before turning from network security mechanisms to the practices that tie all of them together.

Sources