Best Practices for API Rate Limiting in Python

Rate limiting is a budgeting problem wearing an engineering costume. Part of the Making HTTP Requests with the Requests Library guide. A provider hands you a fixed allowance per minute; your job is to spend it evenly, spend all of it, and never spend more. Most builders skip straight to a retry loop and treat the allowance as something you discover by hitting it, which is how a sync job that worked fine on a laptop turns into an IP ban the week you add a second worker. This page covers the design decisions — how big your budget really is, which algorithm to enforce it with, how to let the provider steer you, and what each option costs per month. For the diagnostic side of the problem, debugging 429 Too Many Requests errors is the companion playbook.

Size the budget before you write a limiter

Two numbers govern every outbound integration and they are not the same number. Throughput is requests per second. Concurrency is requests in flight at any instant. Providers publish the first and enforce consequences on the second, and builders routinely tune the wrong one.

Little's law connects them: in-flight requests equal throughput multiplied by latency. If a provider allows 600 requests per minute — 10 per second — and its p95 latency is 180 milliseconds, you need roughly two connections open to saturate the budget. Two. Yet the default deployment for a Python worker fleet is often twelve replicas each holding a pool of ten connections, which is 120 potential in-flight calls against a budget that can absorb two. The fleet does not go faster; it just arrives in bursts and gets rejected.

The inverse case bites just as hard. If the provider is slow — a 2.5 second p95 is normal for report generation, geocoding, or anything LLM-shaped — sustaining that same 10 requests per second demands 25 concurrent calls. Ship four workers with a pool of two and you will use a third of the allowance you already paid for while wondering why the nightly job overruns.

In-flight requests needed to sustain 10 requests per second A bar chart showing that at 100 ms latency one concurrent request saturates a 10 per second budget, rising to 5 at 500 ms, 10 at 1 second and 25 at 2.5 seconds. Sustaining 10 req/s: concurrent calls required 25 15 5 0 1 5 10 25 100 ms 500 ms 1.0 s 2.5 s provider p95 latency

So measure p95 latency for a day, multiply by your target rate, add about 30 percent headroom, and set that number as both your connection pool ceiling and your semaphore size. Doing this before writing any limiter code frequently removes the need for the limiter entirely, and it prevents the related failure of connection pool exhaustion on your own database when a burst of outbound work backs up.

Choose the algorithm your provider actually enforces

Four algorithms dominate, and they behave very differently at the boundary between windows. Match yours to the provider's or you will spend the allowance in a shape it rejects.

Comparison of four rate limiting algorithms A matrix comparing fixed window, sliding log, token bucket and GCRA on burst behaviour, Redis footprint and a recommended verdict. Picking the enforcement algorithm Algorithm Burst behaviour Redis footprint Verdict Fixed window 2x spike at boundary 1 INCR + EXPIRE Avoid Sliding log Exact, no spike ZSET per key Costly Token bucket Burst = capacity 1 hash + Lua In-process GCRA Smooth, bounded 1 timestamp Fleet-wide Verdict assumes a paid third-party API and more than one worker replica.

Fixed windows are what naive implementations reach for and what makes traffic spiky. A counter that resets at the top of each minute lets you send the full allowance in the last second of one window and again in the first second of the next — a 2x burst the provider's own edge will reject even though your arithmetic says you complied. Sliding logs fix the accuracy at the price of storing a timestamp per request, which is fine at 10 requests per second and unaffordable at 500. Token bucket gives you a tunable burst allowance with two floats of state. GCRA — the generic cell rate algorithm, the same maths as a leaky bucket — stores a single timestamp and produces the smoothest arrival pattern of the four, which is why it belongs in Redis.

Pick token bucket inside a process and GCRA across a fleet. The one nuance that matters commercially: some providers price by weight, not by call. GraphQL endpoints charge query points, and LLM providers enforce tokens per minute alongside requests per minute, so your limiter must accept a variable cost per acquisition rather than always decrementing by one. Any implementation that hardcodes a decrement of 1 will need rewriting the day you touch an AI vendor, which is exactly the trap described in controlling LLM API costs in production.

Shape the burst with a token bucket, not a sleep

A hardcoded time.sleep(0.2) between calls is not rate limiting; it is a throughput cap that ignores the time your own code already spent. If the provider call took 180 milliseconds, sleeping another 200 gives you 2.6 requests per second against a budget of 10, so you pay for capacity you never use. A token bucket accounts for elapsed time automatically and lets legitimate bursts through.

Token bucket burst then steady state A bucket refilling at five tokens per second with capacity ten releases ten requests immediately, then one request every two hundred milliseconds. Burst of 10, then a steady 5 req/s refill 5 tokens/s capacity 10 tokens on hand spend 10 at once 1 every 200 ms t = 0 t = 0.6 s t = 1.4 s bucket empty: arrivals pace to the refill rate

The async implementation below is the one to copy. Two details make it production-grade. It releases the lock before sleeping, because holding it across an await asyncio.sleep() serialises every caller into a single queue and destroys the burst you paid for. And it accepts a cost argument, so a weighted endpoint can charge five tokens without a second limiter.

Python
import asyncio
import os
import time


class AsyncTokenBucket:
    """Paces outbound calls to a sustained rate with a bounded burst."""

    def __init__(self, rate: float, capacity: float) -> None:
        self._rate = rate
        self._capacity = capacity
        self._tokens = capacity
        self._updated = time.monotonic()
        self._lock = asyncio.Lock()

    def _refill(self) -> None:
        now = time.monotonic()
        self._tokens = min(self._capacity, self._tokens + (now - self._updated) * self._rate)
        self._updated = now

    def set_rate(self, rate: float) -> None:
        """Retune while running — the header steerer calls this."""
        self._rate = max(rate, 0.1)

    async def acquire(self, cost: float = 1.0) -> None:
        while True:
            async with self._lock:
                self._refill()
                if self._tokens >= cost:
                    self._tokens -= cost
                    return
                wait = (cost - self._tokens) / self._rate
            await asyncio.sleep(wait)


limiter = AsyncTokenBucket(
    rate=float(os.getenv("OUTBOUND_RPS", "5")),
    capacity=float(os.getenv("OUTBOUND_BURST", "10")),
)

Let the response headers steer the limiter

A constant configured rate is always wrong. It is too high the day the provider silently tightens a limit and too low every other day, and nobody ever revisits the environment variable. Close the loop instead: after each response, recompute the target rate from the remaining allowance divided by the seconds left in the window, then multiply by a safety factor around 0.8. The limiter converges on the true budget without a deploy.

Fixed rate versus header-steered rate over one window A line chart of remaining quota across a sixty second window: a fixed ten per second client exhausts its budget at thirty-eight seconds and then collects rejections, while a header-steered client lands on zero exactly at the reset. Quota remaining across one 60 s window 600 300 0 22 s of 429s before reset fixed 10 req/s header-steered 0 s 30 s 60 s (reset)

Two edge cases will break a naive steerer. First, x-ratelimit-reset is an epoch timestamp on some providers and a delta in seconds on others; parse both or you will compute a window of 1.7 billion seconds and throttle yourself to nothing. Second, remaining is computed per edge node, so it can jump upward between responses — never treat a rising value as free capacity, only ever ratchet the rate down within a window and let the reset restore it.

Python
import os
import time

import httpx

SAFETY = float(os.getenv("RATE_LIMIT_SAFETY", "0.8"))
FLOOR_RPS = float(os.getenv("RATE_LIMIT_FLOOR_RPS", "1"))


def seconds_until_reset(raw: str) -> float:
    """Accepts either an epoch timestamp or a delta in seconds."""
    value = float(raw)
    return max(value - time.time(), 1.0) if value > 1e9 else max(value, 1.0)


def steer(bucket, response: httpx.Response) -> None:
    match response.status_code:
        case 429:
            retry_after = response.headers.get("retry-after", "1")
            bucket.set_rate(1.0 / max(float(retry_after), 1.0))
            return
        case code if code >= 500:
            return

    remaining = response.headers.get("x-ratelimit-remaining")
    reset = response.headers.get("x-ratelimit-reset")
    if remaining is None or reset is None:
        return
    try:
        target = (float(remaining) / seconds_until_reset(reset)) * SAFETY
    except ValueError:
        return
    bucket.set_rate(max(target, FLOOR_RPS))

Emit the computed rate as a metric on every adjustment. Without it you cannot tell a healthy limiter from one pinned at its floor, which is the single most useful signal to add when wiring outbound calls into monitoring and logging for Python APIs; a rate field on every request log via structured logging with structlog costs nothing and answers the question instantly.

Share one budget across the whole fleet

An in-process bucket is correct only while exactly one process exists. Scale to four replicas and each one believes it owns the entire allowance, so you emit four times the budget and the provider rejects three quarters of it. Autoscaling makes this worse: the fleet grows precisely when traffic is heavy, multiplying the overage at the worst moment.

The fix is one shared counter, evaluated atomically. GCRA in a Lua script does it in a single round trip against a single key — no read-modify-write race, no ZSET growth, no per-request memory. If you already run Redis for caching API responses, you have the infrastructure.

Three workers sharing one Redis GCRA budget A sequence diagram in which three worker replicas each evaluate the same Lua script against one Redis key; two are allowed immediately and one is told to wait 140 milliseconds. One key, three replicas, no race worker A worker B worker C Redis: one key EVAL gcra, cost 1 allow, wait 0 ms EVAL gcra, cost 5 deny, retry in 140 ms EVAL gcra, cost 1 allow, wait 0 ms Each call is one atomic round trip: about 0.4 ms in-region.
Python
import asyncio
import os
import time

from redis.asyncio import Redis

GCRA = """
local key = KEYS[1]
local now = tonumber(ARGV[1])
local interval = tonumber(ARGV[2])
local tolerance = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])
local tat = tonumber(redis.call('GET', key)) or now
if tat < now then tat = now end
local allow_at = tat - tolerance
if now < allow_at then
  return {0, math.ceil(allow_at - now)}
end
local new_tat = tat + (interval * cost)
redis.call('SET', key, new_tat, 'PX', math.ceil(interval * cost + tolerance) + 1000)
return {1, 0}
"""


class SharedLimiter:
    """Fleet-wide GCRA limiter: one Redis key, one atomic round trip."""

    def __init__(self, client: Redis, key: str, rps: float, burst: int) -> None:
        self._script = client.register_script(GCRA)
        self._key = key
        self._interval_ms = 1000.0 / rps
        self._tolerance_ms = self._interval_ms * burst

    async def acquire(self, cost: float = 1.0) -> None:
        while True:
            allowed, retry_ms = await self._script(
                keys=[self._key],
                args=[time.time() * 1000, self._interval_ms, self._tolerance_ms, cost],
            )
            if allowed:
                return
            await asyncio.sleep(retry_ms / 1000)


def build_limiter() -> SharedLimiter:
    return SharedLimiter(
        Redis.from_url(os.environ["REDIS_URL"]),
        key=os.getenv("RATE_LIMIT_KEY", "outbound:provider"),
        rps=float(os.getenv("OUTBOUND_RPS", "10")),
        burst=int(os.getenv("OUTBOUND_BURST", "10")),
    )

Decide the Redis-down behaviour before you ship, not during the incident. For a paid third-party API, fail open onto a conservative local bucket at roughly a quarter of the fleet rate — a stalled pipeline usually costs more than a handful of rejections. Keep Redis in the same region as the workers; a cross-region round trip turns 0.4 milliseconds of overhead into 35, which at 10 requests per second is six minutes of added wall time per hour.

What rate limiting actually costs

Run the numbers on a job making 5 million outbound calls a month against a provider charging $0.50 per thousand requests. Rejected calls still cost you a TLS handshake, a pool slot, and worker CPU, and many providers bill the attempt.

Monthly waste by limiting strategy Horizontal bars showing 312 dollars of monthly waste from a blind retry loop, 96 dollars from per-worker buckets, and 11 dollars from a shared Redis GCRA limiter. Monthly waste at 5M calls, $0.50 per 1,000 Blind retry loop Per-worker buckets Shared Redis GCRA $312 $96 $11 $0 $150 $300 Waste = rejected calls billed + retry compute + the Redis instance.
StrategyRejection rateMonthly wasteAdded latency
Blind retry loop12.5%$3120 ms
Per-worker buckets3.8%$960 ms
Shared Redis GCRA0.1%$110.4 ms

The shared limiter's $11 is almost entirely the Redis instance itself — a 250 MB managed key store runs $8 to $10 a month and handles 5 million script evaluations without noticing. The 0.4 milliseconds it adds is under half a percent of a 100 millisecond provider call, so throughput is unchanged. Against $312 of billed rejections, the payback is immediate, and the calculation generalises: fold the limiter's overhead into your unit economics using the method in calculating cost per API request. The same shared-state trade-off shows up again when choosing between Redis and in-memory caching for FastAPI.

Failure modes that survive a good limiter

  • Retrying non-retryable statuses. Only 429 and 5xx deserve a retry. Repeating a 401 or 403 burns budget and gets your traffic scored as hostile; fix the credential instead, and see debugging 401 Unauthorized errors.
  • Blocking sleeps inside async code. A time.sleep() in a coroutine freezes every other task on the loop. Use asyncio.sleep() and an async-native client — the reasoning is laid out in httpx vs requests for async.
  • Deterministic backoff across a fleet. Without jitter, every worker retries on the same tick and rebuilds the burst that caused the rejection. Delegate the policy to retrying failed HTTP requests with tenacity rather than hand-rolling it twice.
  • Scheduling every job at minute zero. Cron defaults stack five pipelines onto the same second. Spread start times, or hand pacing to a queue as covered in scheduling data pipelines with cron.
  • Never testing the limiter. Assert the arrival pattern with a fake clock and a mocked transport; mocking external APIs with respx makes a 429-then-recover sequence a three-line fixture.

FAQ

What does an unlimited retry loop cost me at 5M outbound calls a month? About $312 in billed rejections at $0.50 per thousand calls with a 12.5 percent rejection rate, plus the compute for workers sitting in backoff instead of doing billable work. A shared Redis limiter cuts that to roughly $11, almost all of which is the Redis instance. The migration is an afternoon, so it pays back in the first week.

Should I pace my requests or just buy the higher provider tier? Pace first. Shaping is a one-time engineering cost with no recurring spend, and it reveals your true required throughput. Once a properly shaped client saturates its budget for several hours a day, you have hard evidence to price the upgrade against the revenue that traffic produces rather than buying headroom you never use.

How do I rate limit an API that charges by weight instead of by call? Make cost a parameter, not a constant. Estimate the weight before the call — GraphQL query points, prompt tokens, page counts — and pass it to acquire(cost=n). Then reconcile against whatever the response reports as actually consumed, and carry the difference into the next acquisition so a systematic underestimate cannot drift you past the budget.

Does adding a shared limiter risk breaking my pipeline if Redis goes down? Only if you leave the behaviour undefined. Pick fail-open with a conservative local bucket at about a quarter of the fleet rate for third-party APIs, and fail-closed only when exceeding the limit risks account suspension. Keep the acquire() signature identical across both implementations so switching is a constructor change, not a rewrite.

Do the same rules apply to limits on the API I sell? The mechanics do, the incentives flip. As a provider you want limits that protect margin without punishing your best customers, which means per-key budgets tied to tiers and honest retry-after headers. Start from designing API pricing tiers and treat the limiter as a billing control, not just a safety valve.