Rotating API Keys Without Downtime

Sooner or later you have to change a customer's API key. A contractor leaves, a key lands in a public repository, or your own security policy says credentials expire after a year. The naive move — generate a new key, overwrite the old row, email the customer — takes every integration they built down until a human notices and redeploys. This page resolves one decision: whether to swap keys atomically or to run two valid keys side by side, and it gives you the schema, the endpoint and the revocation path to do the second one properly. It is part of the Handling API Authentication in Python guide, which covers issuing and verifying credentials in the first place.

The short version: never swap. Overlap. A key is not a value you replace, it is a row with a lifecycle, and rotation is a state transition that two keys live through together.

Why the atomic swap always costs you an outage

Think about where your key actually lives on the customer's side. It is in a Render environment variable, a GitHub Actions secret, a Lambda config, a .env on someone's laptop, and probably pasted into a Postman collection. There is no single place to update, and no way for you to update any of them. The moment you invalidate the old value, every one of those call sites starts returning 401 — and your support inbox fills with the exact confusion described in debugging 401 Unauthorized API errors.

The cost is not just goodwill. If you bill per successful request, a two-hour outage on a customer doing 40 requests per second is roughly 288,000 unbilled calls, and worse, it teaches their engineering team that your API is fragile. Churn after a self-inflicted auth outage is brutal because the customer has a concrete, shareable story about why you broke them.

The overlap approach removes the cliff entirely. You mint the new key while the old one still works, hand the new value to the customer, and let them migrate call site by call site on their own schedule. The old key retires only when it stops receiving traffic, or when its grace deadline passes — whichever comes first.

Flag-day key swap versus dual-key overlap on a timeline Two timelines. The top one shows an old key ending exactly where a new key begins, with a marked failure point at the boundary. The bottom one shows the old key and new key valid at the same time across a shaded grace window, so no request fails. Two rotation strategies, same calendar Flag-day swap Old key valid New key valid Every deployed client 401s here Dual-key overlap Old key — still valid New key — live Grace window: both accepted Zero failed calls

Store keys so rotation is even possible

You cannot run an overlap window if your database holds one key column on the customer row. Rotation needs keys to be their own table with their own status, and it needs the stored form to be a hash — because the second reason you rotate is that a raw key leaked, and if you store raw keys a database dump leaks all of them at once.

Hashing creates a lookup problem: you cannot index a value you only see hashed at request time. Solve it by splitting the key into a public lookup id and a secret half. The lookup id is indexed and meaningless on its own; the secret half is what you hash and compare. This gives you an O(1) index hit followed by exactly one constant-time comparison, instead of hashing every key in the table.

Use HMAC-SHA-256 with a server-side pepper, not bcrypt. Bcrypt exists to slow down attacks on low-entropy human passwords; a 256-bit random key has no dictionary to attack, and bcrypt's deliberate 50–100 ms cost lands on every single authenticated request. At 200 requests per second that is enough CPU to force a second instance you did not need, which shows up directly in your cost per API request.

Python
# apikeys/mint.py
import hashlib
import hmac
import os
import secrets

KEY_PREFIX = os.getenv("API_KEY_PREFIX", "pk_live")
KEY_PEPPER = os.getenv("API_KEY_PEPPER", "")  # 32+ random bytes, from your secret store


def hash_secret(secret_part: str) -> str:
    if not KEY_PEPPER:
        raise RuntimeError("API_KEY_PEPPER is not set")
    return hmac.new(
        KEY_PEPPER.encode(), secret_part.encode(), hashlib.sha256
    ).hexdigest()


def mint_key() -> tuple[str, str, str]:
    """Return (raw_key, lookup_id, key_hash). Persist only the last two."""
    lookup_id = secrets.token_hex(6)          # 12 public chars, indexed
    secret_part = secrets.token_urlsafe(32)   # 256 bits of entropy
    raw_key = f"{KEY_PREFIX}_{lookup_id}.{secret_part}"
    return raw_key, lookup_id, hash_secret(secret_part)


def split_key(raw_key: str) -> tuple[str, str] | None:
    match raw_key.split(".", 1):
        case [head, secret_part] if head.startswith(f"{KEY_PREFIX}_"):
            return head.removeprefix(f"{KEY_PREFIX}_"), secret_part
        case _:
            return None

The prefix is not decoration. pk_live_ in front of the lookup id lets GitHub's secret scanning and your own log scrubbers recognise your credentials on sight, and it tells a customer at a glance whether they pasted a test key into production.

Hashed API key lookup pipeline A four-stage pipeline: the client header is split at the dot into a public lookup id and a secret, the lookup id hits an index, the secret is HMAC hashed and compared in constant time, and the result branches into a matched customer row or a 401 response. One index hit, one constant-time compare Client header pk_a1b2.secret Split at the dot id + secret Index lookup on lookup id HMAC compare constant time Match: row + tier No match: 401 The raw key exists only in the request, never in your database

The table itself is small and boring, which is the point. Every key is a row; a customer with two live keys has two rows.

Python
# apikeys/schema.py
import asyncio
import os

import asyncpg

DDL = """
CREATE TABLE IF NOT EXISTS api_keys (
    lookup_id      text PRIMARY KEY,
    key_hash       text NOT NULL,
    customer_id    text NOT NULL,
    status         text NOT NULL DEFAULT 'active',
    label          text,
    created_at     timestamptz NOT NULL DEFAULT now(),
    expires_at     timestamptz,
    last_used_at   timestamptz,
    revoked_reason text
);
CREATE INDEX IF NOT EXISTS api_keys_customer_live
    ON api_keys (customer_id) WHERE status IN ('active', 'retiring');
"""


async def main() -> None:
    conn = await asyncpg.connect(os.environ["DATABASE_URL"])
    try:
        await conn.execute(DDL)
    finally:
        await conn.close()


if __name__ == "__main__":
    asyncio.run(main())

The overlap window as a state machine

Three statuses carry the whole design. active means the key is the customer's current credential. retiring means it still authenticates but is on a clock — every response it serves carries a deadline header so a machine, not just a human, can notice. revoked means dead now, no grace, no appeal.

Rotation is one transaction: insert a new active row, flip the previous active row to retiring with an expires_at. A leak skips the middle state entirely and goes straight to revoked. That distinction is the entire reason to model status as a column instead of just deleting rows — planned rotation and incident response need different behaviour from the same table.

API key lifecycle state machine States run left to right from created to active to retiring to revoked, with captions describing what each state does to a request, plus a curved shortcut arrow from active straight to revoked labelled as a leak skipping the grace window. Key lifecycle Leak: skip the grace window created active retiring revoked minted, unused served and metered served + warn header 401 immediately A key reaches revoked when its traffic hits zero, or when a leak forces it

The verification dependency reads the status and decides. Note that a retiring key still returns 200 — it just tells the truth about its deadline in the response headers, so the customer's own monitoring can catch what their inbox missed.

Python
# apikeys/verify.py
import hmac
import os
from datetime import datetime, timezone

from fastapi import HTTPException, Request, Response, status

from .mint import hash_secret, split_key

EXPIRES_HEADER = os.getenv("API_KEY_EXPIRES_HEADER", "X-Api-Key-Expires-At")


async def current_key(request: Request, response: Response) -> dict:
    raw = request.headers.get("authorization", "").removeprefix("Bearer ").strip()
    parts = split_key(raw)
    if parts is None:
        raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Malformed API key")
    lookup_id, secret_part = parts

    row = await request.app.state.pool.fetchrow(
        "SELECT lookup_id, key_hash, customer_id, status, expires_at "
        "FROM api_keys WHERE lookup_id = $1",
        lookup_id,
    )
    if row is None or not hmac.compare_digest(row["key_hash"], hash_secret(secret_part)):
        raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Invalid API key")

    now = datetime.now(timezone.utc)
    match (row["status"], row["expires_at"]):
        case ("revoked", _):
            raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Key revoked")
        case ("retiring", deadline) if deadline is not None and deadline <= now:
            raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Key expired")
        case ("retiring", deadline) if deadline is not None:
            response.headers[EXPIRES_HEADER] = deadline.isoformat()
            response.headers["Warning"] = '299 - "API key retiring, rotation incomplete"'
        case _:
            pass
    return dict(row)

Let the customer rotate themselves

The rotation endpoint is what turns this from an internal runbook into a product feature. A customer who can rotate at 2am without emailing you is a customer who actually rotates, and self-service rotation is table stakes for anyone doing a SOC 2 review of your service. Ship it alongside the rest of your developer portal.

Two rules matter. Authenticate the rotate call with the key being rotated — no separate credential to lose — and cap live keys per customer at two, so nobody accumulates a drawer of forgotten credentials. Return the raw key exactly once, in the response body, and never store it.

Python
# apikeys/routes.py
import os
from datetime import datetime, timedelta, timezone

from fastapi import APIRouter, Depends, HTTPException, Request, status

from .mint import mint_key
from .verify import current_key

router = APIRouter(prefix="/v1/keys", tags=["keys"])
GRACE_HOURS = int(os.getenv("API_KEY_GRACE_HOURS", "168"))
MAX_LIVE_KEYS = int(os.getenv("API_KEY_MAX_LIVE", "2"))


@router.post("/rotate", status_code=status.HTTP_201_CREATED)
async def rotate(request: Request, key: dict = Depends(current_key)) -> dict:
    raw, lookup_id, key_hash = mint_key()
    deadline = datetime.now(timezone.utc) + timedelta(hours=GRACE_HOURS)
    async with request.app.state.pool.acquire() as conn:
        async with conn.transaction():
            live = await conn.fetchval(
                "SELECT count(*) FROM api_keys "
                "WHERE customer_id = $1 AND status IN ('active', 'retiring')",
                key["customer_id"],
            )
            if live >= MAX_LIVE_KEYS:
                raise HTTPException(status.HTTP_409_CONFLICT, "Retire a key first")
            await conn.execute(
                "UPDATE api_keys SET status = 'retiring', expires_at = $2 "
                "WHERE lookup_id = $1 AND status = 'active'",
                key["lookup_id"], deadline,
            )
            await conn.execute(
                "INSERT INTO api_keys (lookup_id, key_hash, customer_id, status) "
                "VALUES ($1, $2, $3, 'active')",
                lookup_id, key_hash, key["customer_id"],
            )
    return {"api_key": raw, "previous_key_expires_at": deadline.isoformat()}

Watch the old key's traffic decay after each rotation — that curve is your real signal for whether the grace window is long enough. Record it from the same event stream you use for logging API usage events to Postgres; a last_used_at touch on the key row costs almost nothing if you batch it.

Traffic decay on a retiring API key A bar chart of the share of requests still using the old key after a rotation: one hundred percent on day zero, sixty-two on day one, twenty-eight on day three, nine on day seven, three on day ten and under one percent on day fourteen. Share of calls still using the old key 100% 50% 0% 100 62 28 9 3 0.4 day 0 day 1 day 3 day 7 day 10 day 14 Days since the customer called /v1/keys/rotate

Revoking a leaked key immediately

Everything above assumes a planned rotation. A leak inverts the priority: breaking the customer's integration is now the correct outcome, because the alternative is an attacker running up their metered usage on your infrastructure. Revocation must be immediate and global.

The trap here is your own cache. Any sane key check caches the verified row — caching API responses with Redis applies just as much to auth lookups, since it is the difference between one database round trip per request and one per minute. A 60-second cache means a revoked key keeps working for up to 60 seconds on every instance. Delete the cache entry in the same call that flips the row, and publish the revocation so instances holding in-process copies drop them too.

Python
# apikeys/revoke.py
import os

import redis.asyncio as redis

CACHE_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
CACHE_PREFIX = os.getenv("API_KEY_CACHE_PREFIX", "apikey:")
REVOKE_CHANNEL = os.getenv("API_KEY_REVOKE_CHANNEL", "key-revocations")


async def revoke_now(pool, lookup_id: str, reason: str) -> None:
    """Kill a key everywhere: database row, shared cache, and in-process copies."""
    await pool.execute(
        "UPDATE api_keys SET status = 'revoked', expires_at = now(), "
        "revoked_reason = $2 WHERE lookup_id = $1",
        lookup_id, reason,
    )
    client = redis.from_url(CACHE_URL)
    try:
        await client.delete(f"{CACHE_PREFIX}{lookup_id}")
        await client.publish(REVOKE_CHANNEL, lookup_id)
    finally:
        await client.aclose()

Emit a structured event on every revocation with the reason and the actor, using the pattern from structured logging with structlog. When a customer asks six months later why their key died, the answer needs to be a log line, not a guess.

When to overlap, and when not to

Overlap is right for essentially every planned rotation. It is wrong when the key is known-compromised, when a customer's contract ends, or when an employee with access to the credential leaves under a cloud. In those cases the failed requests are the feature.

SituationWindowRationale
Scheduled annual rotation30 daysFits a customer sprint cycle
Self-service rotation7 daysDefault; covers a weekend plus slack
Suspicious activity1 hourEnough to redeploy, not to abuse
Confirmed leakNoneRevoke instantly, notify after

Traffic level shifts the default. A customer at ten requests a day may not exercise every code path in a week, so lengthen the window until you have seen zero old-key traffic for at least three of their natural cycles. A customer at a thousand requests a minute proves migration within hours, and holding the old key open beyond that just widens the exposure. Cost is not the constraint here — an extra row and one index entry per customer is free — so pick the window from risk, not from storage.

Migrating from a single plaintext key column

If you already shipped customers.api_key as a plaintext column, the switch takes four steps and no downtime. First, create the api_keys table and start writing new keys to it while verification checks both places, old column first for a miss. Second, backfill: for every existing customer, insert a row with the lookup id derived from the existing key and the hash computed from its secret half — if your current keys have no dot-separated structure, mint fresh keys instead and run the overlap from that moment. Third, announce and rotate: call the rotation flow for each customer, keeping the legacy column valid as the retiring credential. Fourth, once old-key traffic is zero, drop the column.

The same rollout discipline covered in zero-downtime deploys for Python APIs applies: dual-read before you dual-write, and never remove the old path in the same release that adds the new one. Cover both branches with tests before you touch production, using the approach in testing Python APIs with pytest — a rotation bug is an outage that authenticates as your customer.

Builder verdict

Ship the dual-key overlap with a seven-day default and a self-service rotate endpoint, and never write an atomic key swap. The extra work is roughly one table, one endpoint and a status check in the dependency you already have — call it an afternoon. What you get back is the ability to respond to a leaked credential in seconds without arguing about blast radius, a compliance answer that closes enterprise deals, and zero support tickets from planned rotations. The atomic swap looks cheaper only until the first time you use it; one auth outage on a paying customer costs more in refunds, credibility and engineering time than the overlap pattern costs to build twice over. Build it before your first enterprise customer asks, because they will ask, and "we are working on it" is a worse answer than a link to your docs.

FAQ

How long should the grace window be before it becomes a security risk? Seven days is the sweet spot for self-service rotation and thirty for a scheduled policy rotation. Risk scales with the window only if you assume the old key is already compromised — and if you assume that, you should revoke instead of retire. Keep the window, but watch the old key's traffic and close it early once it hits zero.

Does an extra key row per customer cost anything meaningful at scale? No. At 10,000 customers each holding two keys you are storing 20,000 rows of roughly 200 bytes — four megabytes, entirely cached in memory by Postgres. Verification is one index hit either way. The only real cost is the HMAC, which runs in microseconds; if you use bcrypt instead you will pay in extra instances.

What happens to metered billing during the overlap? Bill the customer, not the key. Attribute usage to customer_id so a rotation never splits an invoice or resets a quota. If you also want per-key visibility for the customer's own debugging, log lookup_id alongside customer_id on each usage event and aggregate on the customer for billing.

Can I force rotation on a schedule without breaking customers? Yes, if you announce it in-band. Set expires_at at issue time, return it in the response headers on every call, and send a webhook or email at the 30-, 7- and 1-day marks. Customers who ignore all of that will still break, which is why the header matters: their monitoring can fail the build before your API fails their production.

How do I roll this out without a maintenance window? Deploy verification that accepts both the old and the new storage format first, backfill, then rotate customer by customer, then remove the old path in a later release. That is the same staged pattern used when deprecating an API endpoint without breaking customers — never flip reads and writes in one deploy.

Same track:

Other tracks: