โ† Back to Blog
AI EngineeringLLMsTools

LLM Rate Limits in Production: Token Buckets, Jittered Retries, and Backpressure That Actually Holds

A 429 from your LLM provider is not an error, it is a signal. Here is how to build a client-side limiter, retry policy, and backpressure layer so a traffic spike degrades gracefully instead of cascading.

By Joinloopยท13 October 2026ยท9 min read
ShareLinkedInXReddit
๐Ÿšฆ

Your LLM feature works fine in staging. Then a customer kicks off a batch of 500 documents, your agent fans out into 8 parallel tool calls per request, and the provider starts returning 429 Too Many Requests. Your naive try/except retries immediately. Every worker retries at the same moment. The 429s get worse, latency explodes, and your queue backs up until something falls over.

This is the most common reliability failure in LLM apps, and it is almost entirely avoidable. The fix is not "add a retry decorator." It is a small system with four parts: a client-side limiter, a disciplined retry policy, backpressure, and a fallback path.

Why LLM rate limits are different from normal API limits

Most REST APIs limit requests per second. LLM providers typically limit along several dimensions at once:

  • Requests per minute (RPM)
  • Input tokens per minute (ITPM) and output tokens per minute (OTPM), or a combined tokens-per-minute (TPM) figure, depending on provider
  • Concurrent requests on some tiers or endpoints
  • Daily or monthly spend caps

Two things make this tricky:

  1. Token cost is unknown up front. You know your input size, but output length is only bounded by max_tokens. Providers often reserve capacity against max_tokens, so setting it to 8,000 "just in case" can exhaust your quota faster than the tokens you actually generate.
  2. A single long request can consume as much quota as hundreds of short ones. Counting requests alone gives you a limiter that looks healthy while you are over budget.

Also separate two failure classes. A 429 means you exceeded your quota. A 5xx or "overloaded" response (Anthropic uses 529 for this) means the provider is under pressure. Both are retryable, but the right response differs: for 429 you should slow your own rate; for overload you should back off and consider a fallback model.

Check your provider's current docs for exact limit semantics and header names. They change, and tiers differ by account.

The four-layer design

Layer Job Failure it prevents
Client-side limiter Stay under quota before sending Self-inflicted 429s
Retry with jittered backoff Recover from transient errors Lost requests, thundering herd
Backpressure (bounded queue + concurrency cap) Refuse work you cannot do Unbounded memory, cascading latency
Fallback Keep serving when the primary is degraded Hard outages

Layer 1: A token-aware limiter

A token bucket refills at a steady rate and lets you burst up to its capacity. You need one bucket for requests and one for tokens. The limiter below is async-safe and takes an estimated token cost per call.

import asyncio
import time


class TokenBucket:
    """Refills `rate` units per second up to `capacity`."""

    def __init__(self, rate: float, capacity: float):
        self.rate = rate
        self.capacity = capacity
        self.tokens = capacity
        self.updated = time.monotonic()
        self._lock = asyncio.Lock()

    async def acquire(self, cost: float = 1.0) -> None:
        if cost > self.capacity:
            raise ValueError("cost exceeds bucket capacity")
        while True:
            async with self._lock:
                now = time.monotonic()
                self.tokens = min(
                    self.capacity,
                    self.tokens + (now - self.updated) * self.rate,
                )
                self.updated = now
                if self.tokens >= cost:
                    self.tokens -= cost
                    return
                wait = (cost - self.tokens) / self.rate
            await asyncio.sleep(wait)


class LLMLimiter:
    def __init__(self, rpm: int, tpm: int, max_concurrency: int):
        self.requests = TokenBucket(rate=rpm / 60, capacity=rpm / 6)   # ~10s burst
        self.tokens = TokenBucket(rate=tpm / 60, capacity=tpm / 6)
        self.sem = asyncio.Semaphore(max_concurrency)

    async def __aenter__(self):
        await self.sem.acquire()
        return self

    async def reserve(self, est_tokens: int) -> None:
        await self.requests.acquire(1)
        await self.tokens.acquire(est_tokens)

    async def __aexit__(self, *exc):
        self.sem.release()

Set your configured limits to roughly 80% of the provider's published numbers. The headroom absorbs estimation error and other services sharing the same API key.

Estimate tokens as input_tokens + max_tokens. Use your provider's token-counting endpoint or a local tokenizer for the input side. Overestimating slightly is safe; underestimating gets you throttled.

Layer 2: Retries that do not stampede

The rules:

  1. Only retry what is retryable. 429, 408, 5xx, and connection errors yes. 400, 401, 403, and 404 never. A 400 for an oversized context will fail identically every time.
  2. Honour Retry-After when the provider sends it. It is the provider telling you exactly how long to wait.
  3. Exponential backoff with full jitter otherwise. Without jitter, 200 clients that failed together retry together.
  4. Cap attempts and total elapsed time. An unbounded retry loop is an outage amplifier.
import random
import httpx

RETRYABLE = {408, 409, 429, 500, 502, 503, 504, 529}


def backoff_delay(attempt: int, base: float = 0.5, cap: float = 30.0) -> float:
    # "Full jitter": uniform between 0 and the exponential ceiling
    return random.uniform(0, min(cap, base * (2 ** attempt)))


async def call_with_retries(send, limiter: LLMLimiter, est_tokens: int,
                            max_attempts: int = 5, deadline_s: float = 60.0):
    start = time.monotonic()
    last_exc = None
    for attempt in range(max_attempts):
        if time.monotonic() - start > deadline_s:
            break
        async with limiter:
            await limiter.reserve(est_tokens)
            try:
                resp = await send()
            except (httpx.ConnectError, httpx.ReadTimeout) as e:
                last_exc, delay = e, backoff_delay(attempt)
            else:
                if resp.status_code < 400:
                    return resp
                if resp.status_code not in RETRYABLE:
                    resp.raise_for_status()          # fail fast
                retry_after = resp.headers.get("retry-after")
                delay = float(retry_after) if retry_after else backoff_delay(attempt)
                last_exc = httpx.HTTPStatusError(
                    "retryable", request=resp.request, response=resp
                )
        # Sleep OUTSIDE the semaphore so we don't hold a concurrency slot
        await asyncio.sleep(delay)
    raise last_exc or TimeoutError("retry deadline exceeded")

Note the sleep happens after leaving the async with limiter block. Holding a concurrency slot while sleeping starves healthy requests and is a classic self-inflicted bottleneck.

If you use the official OpenAI or Anthropic SDKs, they already retry a couple of times with backoff by default (check the max_retries option in each). That is useful, but it is invisible to your limiter. Either disable SDK retries and own the policy, or keep them low and treat your layer as the outer loop. Stacked retries multiply: 3 SDK retries inside 5 app retries is 15 attempts per request.

Layer 3: Backpressure, or how to say no

A limiter makes callers wait. If callers keep arriving faster than you can serve, the wait grows without bound. You need a bounded queue and a clear rejection path.

class Overloaded(Exception):
    pass


class BoundedGate:
    def __init__(self, max_waiting: int = 100):
        self.max_waiting = max_waiting
        self.waiting = 0

    async def run(self, coro_fn, *, timeout_s: float = 20.0):
        if self.waiting >= self.max_waiting:
            raise Overloaded("queue full")      # map to HTTP 429/503 upstream
        self.waiting += 1
        try:
            return await asyncio.wait_for(coro_fn(), timeout=timeout_s)
        finally:
            self.waiting -= 1

Decisions this forces you to make deliberately:

  • Interactive traffic vs. batch traffic. Give them separate limiters. A nightly backfill should never be able to starve a user waiting on a chat response. Better still, route batch work to the provider's batch API if latency is not a concern (see our post on LLM batch API cost optimization).
  • What the user sees. "Try again in a few seconds" with a real Retry-After of your own beats a spinner that hangs for 90 seconds.
  • Shed load by priority. When the queue is full, drop low-priority work first.

Layer 4: Fallbacks

When the primary model is overloaded or you are repeatedly throttled, degrade rather than fail:

Fallback Trade-off
Same model, different region or deployment Best quality parity; needs extra capacity provisioned
Smaller, faster model from the same provider Lower quality; usually separate or higher limits
Different provider via a gateway Prompt and tool-schema differences to test
Cached or precomputed answer Fresh only if your cache policy allows
Graceful "unavailable" message Honest, zero cost

Use a circuit breaker so you stop hammering a degraded endpoint: after N failures in a window, route to the fallback for a cooldown, then probe with a single request. Whatever fallback you choose, log which path served each request so evals and cost reports do not silently mix model outputs.

A quick pre-launch checklist

  • Limits configured at โ‰ค80% of provider quota, per API key and per model
  • Token estimate includes max_tokens, and max_tokens is set to a realistic value
  • Retries capped by attempts and wall-clock deadline
  • Retry-After honoured; full jitter otherwise
  • No sleeping while holding a concurrency slot
  • SDK-level retries and app-level retries reconciled
  • Bounded queue with a defined rejection response
  • Interactive and batch traffic isolated
  • Fallback path tested, with the served-by model logged
  • Dashboards for 429 rate, queue depth, and retry count

Common pitfalls

Per-process limiters on a multi-replica service. Ten pods each allowing 1,000 RPM is 10,000 RPM against a 1,000 RPM quota. Either divide the quota by replica count or move the limiter state to Redis (a Lua-scripted token bucket is the usual approach).

Retrying non-idempotent tool calls. Retrying the LLM call is safe. Retrying an agent step that already sent an email is not. Retry at the model-call boundary, not around whole agent turns. For the latter, see our post on durable execution and idempotency for agents.

Streaming responses and mid-stream failures. A stream can fail after you have already shown the user partial output. Decide whether to restart silently, resume, or surface the error. Do not blindly retry a stream you have already forwarded.

Treating 429 as a bug. A rising 429 rate is capacity-planning data. Alert on it, and request a quota increase before launch rather than during the incident.

Ignoring cached-token behaviour. Prompt caching can change how input tokens count toward limits on some providers. Read the docs for your provider, since this affects how aggressively you can safely send.

Next steps

  1. Add a metric for 429s, retries, and queue depth today. You cannot tune what you cannot see.
  2. Replace any bare retry decorator with the policy above, and write a test that simulates 429 with and without Retry-After.
  3. Run a load test at 2x expected peak and confirm requests are rejected quickly rather than queued slowly.
  4. Wire in one fallback path and drill it before you need it.

If you want to build and harden production LLM systems with a cohort of engineers shipping real projects, take a look at the Joinloop AI Engineering cohort.

ShareLinkedInXReddit

Want to build this yourself?

Join a live cohort and go from reading about AI to shipping it, with practitioners who do it daily.