LLM Rate Limits in Production: Token Buckets, Jittered Retries, and Backpressure That Actually Holds
A 429 from your LLM provider is not an error, it is a signal. Here is how to build a client-side limiter, retry policy, and backpressure layer so a traffic spike degrades gracefully instead of cascading.
Your LLM feature works fine in staging. Then a customer kicks off a batch of 500 documents, your agent fans out into 8 parallel tool calls per request, and the provider starts returning 429 Too Many Requests. Your naive try/except retries immediately. Every worker retries at the same moment. The 429s get worse, latency explodes, and your queue backs up until something falls over.
This is the most common reliability failure in LLM apps, and it is almost entirely avoidable. The fix is not "add a retry decorator." It is a small system with four parts: a client-side limiter, a disciplined retry policy, backpressure, and a fallback path.
Why LLM rate limits are different from normal API limits
Most REST APIs limit requests per second. LLM providers typically limit along several dimensions at once:
- Requests per minute (RPM)
- Input tokens per minute (ITPM) and output tokens per minute (OTPM), or a combined tokens-per-minute (TPM) figure, depending on provider
- Concurrent requests on some tiers or endpoints
- Daily or monthly spend caps
Two things make this tricky:
- Token cost is unknown up front. You know your input size, but output length is only bounded by
max_tokens. Providers often reserve capacity againstmax_tokens, so setting it to 8,000 "just in case" can exhaust your quota faster than the tokens you actually generate. - A single long request can consume as much quota as hundreds of short ones. Counting requests alone gives you a limiter that looks healthy while you are over budget.
Also separate two failure classes. A 429 means you exceeded your quota. A 5xx or "overloaded" response (Anthropic uses 529 for this) means the provider is under pressure. Both are retryable, but the right response differs: for 429 you should slow your own rate; for overload you should back off and consider a fallback model.
Check your provider's current docs for exact limit semantics and header names. They change, and tiers differ by account.
The four-layer design
| Layer | Job | Failure it prevents |
|---|---|---|
| Client-side limiter | Stay under quota before sending | Self-inflicted 429s |
| Retry with jittered backoff | Recover from transient errors | Lost requests, thundering herd |
| Backpressure (bounded queue + concurrency cap) | Refuse work you cannot do | Unbounded memory, cascading latency |
| Fallback | Keep serving when the primary is degraded | Hard outages |
Layer 1: A token-aware limiter
A token bucket refills at a steady rate and lets you burst up to its capacity. You need one bucket for requests and one for tokens. The limiter below is async-safe and takes an estimated token cost per call.
import asyncio
import time
class TokenBucket:
"""Refills `rate` units per second up to `capacity`."""
def __init__(self, rate: float, capacity: float):
self.rate = rate
self.capacity = capacity
self.tokens = capacity
self.updated = time.monotonic()
self._lock = asyncio.Lock()
async def acquire(self, cost: float = 1.0) -> None:
if cost > self.capacity:
raise ValueError("cost exceeds bucket capacity")
while True:
async with self._lock:
now = time.monotonic()
self.tokens = min(
self.capacity,
self.tokens + (now - self.updated) * self.rate,
)
self.updated = now
if self.tokens >= cost:
self.tokens -= cost
return
wait = (cost - self.tokens) / self.rate
await asyncio.sleep(wait)
class LLMLimiter:
def __init__(self, rpm: int, tpm: int, max_concurrency: int):
self.requests = TokenBucket(rate=rpm / 60, capacity=rpm / 6) # ~10s burst
self.tokens = TokenBucket(rate=tpm / 60, capacity=tpm / 6)
self.sem = asyncio.Semaphore(max_concurrency)
async def __aenter__(self):
await self.sem.acquire()
return self
async def reserve(self, est_tokens: int) -> None:
await self.requests.acquire(1)
await self.tokens.acquire(est_tokens)
async def __aexit__(self, *exc):
self.sem.release()
Set your configured limits to roughly 80% of the provider's published numbers. The headroom absorbs estimation error and other services sharing the same API key.
Estimate tokens as input_tokens + max_tokens. Use your provider's token-counting endpoint or a local tokenizer for the input side. Overestimating slightly is safe; underestimating gets you throttled.
Layer 2: Retries that do not stampede
The rules:
- Only retry what is retryable. 429, 408, 5xx, and connection errors yes. 400, 401, 403, and 404 never. A 400 for an oversized context will fail identically every time.
- Honour
Retry-Afterwhen the provider sends it. It is the provider telling you exactly how long to wait. - Exponential backoff with full jitter otherwise. Without jitter, 200 clients that failed together retry together.
- Cap attempts and total elapsed time. An unbounded retry loop is an outage amplifier.
import random
import httpx
RETRYABLE = {408, 409, 429, 500, 502, 503, 504, 529}
def backoff_delay(attempt: int, base: float = 0.5, cap: float = 30.0) -> float:
# "Full jitter": uniform between 0 and the exponential ceiling
return random.uniform(0, min(cap, base * (2 ** attempt)))
async def call_with_retries(send, limiter: LLMLimiter, est_tokens: int,
max_attempts: int = 5, deadline_s: float = 60.0):
start = time.monotonic()
last_exc = None
for attempt in range(max_attempts):
if time.monotonic() - start > deadline_s:
break
async with limiter:
await limiter.reserve(est_tokens)
try:
resp = await send()
except (httpx.ConnectError, httpx.ReadTimeout) as e:
last_exc, delay = e, backoff_delay(attempt)
else:
if resp.status_code < 400:
return resp
if resp.status_code not in RETRYABLE:
resp.raise_for_status() # fail fast
retry_after = resp.headers.get("retry-after")
delay = float(retry_after) if retry_after else backoff_delay(attempt)
last_exc = httpx.HTTPStatusError(
"retryable", request=resp.request, response=resp
)
# Sleep OUTSIDE the semaphore so we don't hold a concurrency slot
await asyncio.sleep(delay)
raise last_exc or TimeoutError("retry deadline exceeded")
Note the sleep happens after leaving the async with limiter block. Holding a concurrency slot while sleeping starves healthy requests and is a classic self-inflicted bottleneck.
If you use the official OpenAI or Anthropic SDKs, they already retry a couple of times with backoff by default (check the max_retries option in each). That is useful, but it is invisible to your limiter. Either disable SDK retries and own the policy, or keep them low and treat your layer as the outer loop. Stacked retries multiply: 3 SDK retries inside 5 app retries is 15 attempts per request.
Layer 3: Backpressure, or how to say no
A limiter makes callers wait. If callers keep arriving faster than you can serve, the wait grows without bound. You need a bounded queue and a clear rejection path.
class Overloaded(Exception):
pass
class BoundedGate:
def __init__(self, max_waiting: int = 100):
self.max_waiting = max_waiting
self.waiting = 0
async def run(self, coro_fn, *, timeout_s: float = 20.0):
if self.waiting >= self.max_waiting:
raise Overloaded("queue full") # map to HTTP 429/503 upstream
self.waiting += 1
try:
return await asyncio.wait_for(coro_fn(), timeout=timeout_s)
finally:
self.waiting -= 1
Decisions this forces you to make deliberately:
- Interactive traffic vs. batch traffic. Give them separate limiters. A nightly backfill should never be able to starve a user waiting on a chat response. Better still, route batch work to the provider's batch API if latency is not a concern (see our post on LLM batch API cost optimization).
- What the user sees. "Try again in a few seconds" with a real
Retry-Afterof your own beats a spinner that hangs for 90 seconds. - Shed load by priority. When the queue is full, drop low-priority work first.
Layer 4: Fallbacks
When the primary model is overloaded or you are repeatedly throttled, degrade rather than fail:
| Fallback | Trade-off |
|---|---|
| Same model, different region or deployment | Best quality parity; needs extra capacity provisioned |
| Smaller, faster model from the same provider | Lower quality; usually separate or higher limits |
| Different provider via a gateway | Prompt and tool-schema differences to test |
| Cached or precomputed answer | Fresh only if your cache policy allows |
| Graceful "unavailable" message | Honest, zero cost |
Use a circuit breaker so you stop hammering a degraded endpoint: after N failures in a window, route to the fallback for a cooldown, then probe with a single request. Whatever fallback you choose, log which path served each request so evals and cost reports do not silently mix model outputs.
A quick pre-launch checklist
- Limits configured at โค80% of provider quota, per API key and per model
- Token estimate includes
max_tokens, andmax_tokensis set to a realistic value - Retries capped by attempts and wall-clock deadline
-
Retry-Afterhonoured; full jitter otherwise - No sleeping while holding a concurrency slot
- SDK-level retries and app-level retries reconciled
- Bounded queue with a defined rejection response
- Interactive and batch traffic isolated
- Fallback path tested, with the served-by model logged
- Dashboards for 429 rate, queue depth, and retry count
Common pitfalls
Per-process limiters on a multi-replica service. Ten pods each allowing 1,000 RPM is 10,000 RPM against a 1,000 RPM quota. Either divide the quota by replica count or move the limiter state to Redis (a Lua-scripted token bucket is the usual approach).
Retrying non-idempotent tool calls. Retrying the LLM call is safe. Retrying an agent step that already sent an email is not. Retry at the model-call boundary, not around whole agent turns. For the latter, see our post on durable execution and idempotency for agents.
Streaming responses and mid-stream failures. A stream can fail after you have already shown the user partial output. Decide whether to restart silently, resume, or surface the error. Do not blindly retry a stream you have already forwarded.
Treating 429 as a bug. A rising 429 rate is capacity-planning data. Alert on it, and request a quota increase before launch rather than during the incident.
Ignoring cached-token behaviour. Prompt caching can change how input tokens count toward limits on some providers. Read the docs for your provider, since this affects how aggressively you can safely send.
Next steps
- Add a metric for 429s, retries, and queue depth today. You cannot tune what you cannot see.
- Replace any bare retry decorator with the policy above, and write a test that simulates 429 with and without
Retry-After. - Run a load test at 2x expected peak and confirm requests are rejected quickly rather than queued slowly.
- Wire in one fallback path and drill it before you need it.
If you want to build and harden production LLM systems with a cohort of engineers shipping real projects, take a look at the Joinloop AI Engineering cohort.
Want to build this yourself?
Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.