Retrying Failed HTTP Requests with tenacity Without Amplifying the Outage

A naive retry loop is the fastest way to turn a supplier's five-second blip into your own thirty-minute incident. Three retries per call sounds harmless until every worker in your fleet does it at the same instant against an already-struggling upstream, and you have quietly tripled the load on the exact service that just told you it could not cope. This page resolves one decision: what your retry policy should actually contain when the call is a paid, customer-facing dependency. Part of the Making HTTP Requests with the Requests Library guide.

The short version: use tenacity, retry only on transport errors and a narrow allowlist of status codes, wait with full jitter, cap the whole operation with a deadline rather than an attempt count, attach an idempotency key to every mutating call, and put a circuit breaker in front of the retry so a sustained outage costs you one request per cooldown window instead of thousands.

Which failures are actually worth retrying

Retrying is only correct when the failure is transient and the request is safe to repeat. Every other retry burns money and latency for nothing. A 400 will be a 400 on the fourth attempt too; retrying it just multiplies your egress bill and pushes the eventual error to your customer four seconds late instead of one.

The nuance sits on 500 and on read timeouts. Both mean the server may have already applied your write before failing to tell you about it. Retrying a POST /charges after a read timeout can double-bill a customer, which is a refund, a support ticket, and a chargeback fee — far more expensive than the failed request. So the rule is not "retry 5xx", it is "retry 5xx when the call is idempotent, and make mutating calls idempotent on purpose". A 401 is never a retry candidate either; if you see one intermittently you have a token refresh bug, and debugging 401 unauthorized errors is the right thread to pull instead.

Backoff with real jitter, not decorative jitter

Exponential backoff spaces attempts out. Jitter is what stops your workers from synchronising. Without it, fifty containers that all failed at the same second retry at the same second, again at second three, again at second seven — a self-inflicted load spike arriving exactly when the upstream is trying to recover. This is the same failure shape covered in best practices for API rate limiting, and the fix is identical: randomise across the whole wait window, not a slice of it.

Read tenacity's built-in helper carefully. wait_exponential_jitter(initial, max, exp_base, jitter) computes the exponential delay and then adds a random value between zero and jitter. With jitter=1 on a delay that has already grown to eight seconds, your fleet is still spread across a one-second band — barely spread at all. What you want is full jitter: pick uniformly from zero to the current ceiling, so attempt five lands anywhere in an eight-second window and the herd genuinely disperses.

Here is the policy I ship. It uses httpx because new outbound code should be async — the reasoning is in httpx vs requests for async — and it honours Retry-After when the server sends one, because the server's number always beats your guess.

import logging
import os
import random
import httpx
from tenacity import (
    AsyncRetrying,
    RetryError,
    retry_if_exception_type,
    stop_after_attempt,
    stop_after_delay,
)

logger = logging.getLogger(__name__)

RETRYABLE_STATUS = {408, 425, 429, 500, 502, 503, 504}
BASE_DELAY = float(os.getenv("RETRY_BASE_DELAY", "0.5"))
MAX_DELAY = float(os.getenv("RETRY_MAX_DELAY", "8"))
TOTAL_BUDGET = float(os.getenv("RETRY_TOTAL_BUDGET", "20"))


class RetryableStatus(Exception):
    def __init__(self, status_code: int, retry_after: float | None) -> None:
        super().__init__(f"upstream returned {status_code}")
        self.status_code = status_code
        self.retry_after = retry_after


def full_jitter(retry_state) -> float:
    """Uniform pick from 0..ceiling, or the server's Retry-After if larger."""
    ceiling = min(MAX_DELAY, BASE_DELAY * 2 ** (retry_state.attempt_number - 1))
    delay = random.uniform(0, ceiling)
    exc = retry_state.outcome.exception() if retry_state.outcome else None
    if isinstance(exc, RetryableStatus) and exc.retry_after:
        return max(delay, exc.retry_after)
    return delay


def parse_retry_after(response: httpx.Response) -> float | None:
    raw = response.headers.get("retry-after")
    match raw:
        case None:
            return None
        case value if value.isdigit():
            return float(value)
        case _:
            return None


async def call_upstream(client: httpx.AsyncClient, path: str) -> dict:
    retryer = AsyncRetrying(
        retry=retry_if_exception_type((httpx.TransportError, RetryableStatus)),
        wait=full_jitter,
        stop=stop_after_attempt(5) | stop_after_delay(TOTAL_BUDGET),
        reraise=True,
    )
    async for attempt in retryer:
        with attempt:
            response = await client.get(path)
            if response.status_code in RETRYABLE_STATUS:
                raise RetryableStatus(
                    response.status_code, parse_retry_after(response)
                )
            response.raise_for_status()
            return response.json()
    raise RetryError(None)

stop_after_attempt(5) | stop_after_delay(TOTAL_BUDGET) is the important line. Either condition ends the loop, so a slow upstream cannot stretch one call to forty seconds just because five attempts were technically allowed.

Per-attempt timeouts versus the total budget

Most retry incidents I have debugged were really timeout incidents. The team set a generous thirty-second client timeout, then wrapped it in three retries, and produced a worst case of ninety-plus seconds per inbound request. Their own web server had a sixty-second worker timeout, so the request died anyway — after occupying a worker for a full minute and holding a database connection the whole time. Under load that pattern eats your process pool and takes down endpoints that have nothing to do with the failing supplier.

Set two limits and treat them as different things. The per-attempt timeout should be tight and split by phase: connect timeouts belong near two seconds because a healthy TCP handshake is milliseconds, while read timeouts should sit just above the upstream's genuine p99. The total budget is the number that protects your service, and it must be smaller than whatever your caller will tolerate.

import os
import httpx

def build_client() -> httpx.AsyncClient:
    timeout = httpx.Timeout(
        connect=float(os.getenv("HTTP_CONNECT_TIMEOUT", "2.0")),
        read=float(os.getenv("HTTP_READ_TIMEOUT", "5.0")),
        write=float(os.getenv("HTTP_WRITE_TIMEOUT", "5.0")),
        pool=float(os.getenv("HTTP_POOL_TIMEOUT", "1.0")),
    )
    limits = httpx.Limits(
        max_connections=int(os.getenv("HTTP_MAX_CONNECTIONS", "50")),
        max_keepalive_connections=int(os.getenv("HTTP_MAX_KEEPALIVE", "20")),
    )
    return httpx.AsyncClient(
        base_url=os.environ["UPSTREAM_BASE_URL"],
        timeout=timeout,
        limits=limits,
        headers={"Authorization": f"Bearer {os.environ['UPSTREAM_API_KEY']}"},
    )

Note the pool timeout. When retries pile up, every waiting attempt holds a connection slot; without a pool timeout your coroutines block invisibly on connection acquisition and the symptom looks like an upstream problem when it is yours.

Idempotency keys make writes retryable

Once retries are safe for reads, the remaining risk is writes. If you POST a charge and the read times out, you do not know whether the charge happened. Both choices are bad: retry and risk double-charging, or give up and risk a silent success your database never recorded.

An idempotency key removes the guess. Generate a stable key derived from the business event — not from the attempt — send it on every attempt, and the upstream returns the original result instead of performing the work twice. Stripe, and most payment or messaging APIs worth integrating, support this header. The mirror image of the pattern, applied to traffic you receive, is covered in building an idempotent webhook receiver.

import hashlib
import json
import os
import httpx

IDEMPOTENCY_HEADER = os.getenv("IDEMPOTENCY_HEADER", "Idempotency-Key")


def idempotency_key(event_id: str, payload: dict) -> str:
    """Stable across retries, unique per business event."""
    body = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256(f"{event_id}:{body}".encode()).hexdigest()
    return digest[:40]


async def create_charge(
    client: httpx.AsyncClient, event_id: str, payload: dict
) -> dict:
    response = await client.post(
        "/charges",
        json=payload,
        headers={IDEMPOTENCY_HEADER: idempotency_key(event_id, payload)},
    )
    response.raise_for_status()
    return response.json()

Compute the key once, at the top of the operation, and pass it down. If you generate it inside the retry wrapper you have built an expensive random string generator that protects nothing.

Circuit breaking: the cap on your outage bill

Retries assume the failure is transient. When it is not — a supplier is down for twenty minutes — retries become a fixed multiplier on wasted spend and latency. A breaker sits in front of the retry and short-circuits calls once the failure rate crosses a threshold, so a sustained outage costs you one probe request per cooldown window rather than five attempts per user request. At 200 requests per minute with five attempts each, twenty minutes of blind retrying is 20,000 wasted calls; a breaker with a thirty-second cooldown makes it forty. If those calls are metered, the difference lands straight on your invoice, which is exactly the arithmetic in calculating cost per API request.

import os
import time
from dataclasses import dataclass, field


class BreakerOpen(Exception):
    """Raised without touching the network. Never retry this."""


@dataclass
class CircuitBreaker:
    threshold: int = int(os.getenv("BREAKER_THRESHOLD", "5"))
    cooldown: float = float(os.getenv("BREAKER_COOLDOWN", "30"))
    failures: int = 0
    opened_at: float | None = field(default=None)

    def before_call(self) -> None:
        if self.opened_at is None:
            return
        if time.monotonic() - self.opened_at < self.cooldown:
            raise BreakerOpen("upstream circuit is open")
        self.opened_at = None  # half-open: allow one probe through

    def record(self, ok: bool) -> None:
        if ok:
            self.failures = 0
            self.opened_at = None
        else:
            self.failures += 1
            if self.failures >= self.threshold:
                self.opened_at = time.monotonic()

Wire before_call() at the top of your client method and record() after it, outside the tenacity loop. BreakerOpen must not be in your retry_if_exception_type list — a breaker you retry through is just a slower failure.

When tenacity is the right tool, and when it is not

Reach for tenacity as soon as the call is customer-facing or metered. Its value is that policy lives in one declarative object you can unit-test with respx mocking rather than scattered time.sleep() calls nobody dares touch. Skip it when retrying cannot help: a long-running job that already runs on a queue should let Celery handle redelivery, because an in-process retry there just holds a worker hostage. Skip it too on hot paths where the caller's own deadline is shorter than one backoff step — a 200ms budget has no room for a retry, so fail fast and serve a cached value instead.

Migrating off a hand-rolled retry loop

Five steps, and you can do them in one afternoon. First, add tenacity and httpx and pull every retry constant into environment variables so staging and production can differ. Second, wrap your existing client call in AsyncRetrying with reraise=True and no retry predicate — behaviour is unchanged, which makes the diff safe to merge. Third, narrow the predicate to transport errors plus your status allowlist, and delete the old except Exception: continue. Fourth, replace the attempt counter with stop_after_attempt(n) | stop_after_delay(budget) and tighten the per-attempt timeouts underneath it. Fifth, add the idempotency key to every mutating call and only then enable retries on those paths.

Instrument as you go. Emit attempt_number, the exception class, and the chosen sleep on every retry through structured logging, and put a counter on retries-per-successful-call. If that ratio drifts above roughly 0.1 you are not resilient, you are papering over an upstream problem — or over your own quota, in which case debugging 429 errors is the more useful page.

Builder verdict

tenacity wins, and it is not close. The alternatives either cover a fraction of the problem or hand you a loop you will be too scared to modify six months from now. What actually matters is the shape of the policy, not the library: retry a narrow allowlist, wait with full jitter, bound the whole operation with a deadline instead of a count, make writes idempotent before you let them retry, and put a breaker in front so a supplier's bad day does not multiply your bill. That configuration is roughly forty lines and it converts the most common cause of paging incidents in small API businesses — a dependency wobbles, retries amplify it, everything falls over — into a bounded, observable, cheap failure. If you are integrating a metered upstream such as an LLM provider, pair it with the spend controls in controlling LLM API costs in production; retries are a cost multiplier and deserve a budget line of their own.

FAQ

How much does retrying add to my upstream bill? Model it as your failure rate times your average attempt count. At a 1% failure rate with three attempts you pay for about 2% more calls, which is noise. At a 5% failure rate the same policy costs 10% more, and if the upstream charges per call that is a real margin hit — which is the point at which a circuit breaker stops being optional.

Should I retry inside my API handler or push the work to a queue? If the caller is waiting on the response and your total budget fits inside their timeout, retry inline. If the operation can complete later, enqueue it: a background worker can afford minutes of backoff without holding a web worker, a database connection, and a customer's browser tab open.

Does a retry policy break when I rotate API keys? It can, badly. A stale key produces 401 or 403, and if those are in your retry set you will hammer the upstream with doomed calls and possibly get the whole key blocked. Keep 4xx auth errors out of the retry predicate and let them fail loudly during a rotation.

What is the migration risk of adding tenacity to a live service? Low if you stage it. Ship the wrapper with the old behaviour first, then narrow the predicate, then add the deadline. The one genuinely risky step is enabling retries on writes, so do that only after idempotency keys are live and verified against the upstream's test environment.

How do I stop retries hiding a real outage from my alerts? Alert on the retry rate, not just on final errors. A service that succeeds after four attempts is failing — it is just failing quietly and burning latency budget. Track retries per successful call and page when it crosses your threshold, before customers notice the slowdown.

Related

• Making HTTP Requests with the Requests Library — the parent guide covering sessions, headers and connection reuse.
• Debugging 429 Too Many Requests Errors — read this when your retries are the cause rather than the cure.
• httpx vs requests for Async — why the client in these snippets is async and what it costs to switch.
• Building an Idempotent Webhook Receiver — the same idempotency contract applied to traffic you receive.
• Mocking External APIs with respx — how to unit-test a retry policy without waiting on real backoff.