Building an Idempotent Webhook Receiver

Every webhook provider you will integrate — Stripe, GitHub, Shopify, Paddle, Slack — promises at least once delivery. None of them promise exactly once, and none of them promise order. That means your receiver will eventually be handed the same payment_intent.succeeded twice, and it will be handed a subscription.updated that describes an older state than the one you already stored. If the handler is not idempotent, you refund twice, email twice, or downgrade a paying customer back to free. This page shows the four mechanics that make a receiver safe: an event-id ledger, an atomic claim, versioned state transitions, and a replay tool. It is part of Processing Webhooks with Python, which covers the surrounding receive-verify-enqueue shape.

The recommendation up front: put the deduplication ledger in the same Postgres database as your business data and claim rows with INSERT ... ON CONFLICT DO NOTHING. Redis is faster and wrong for this, because a dedup marker that survives a crash but a state write that does not is worse than no dedup at all.

Why duplicates and reordering are the normal case

Duplicates are not an exotic failure. A provider retries whenever it does not see a 2xx inside its timeout — typically 10 to 30 seconds. Your handler can commit the database write, take 12 seconds to call a slow downstream API, and then get killed by the platform's request timeout before the response flushes. From the provider's point of view nothing happened, so it redelivers. You now have one committed write and one pending redelivery of the same event.

Reordering is just as ordinary. Providers fan delivery out across workers, so two events emitted 40 milliseconds apart travel independent paths. Add a retry to the first one and the second arrives first. Stripe explicitly documents that it does not guarantee ordering, and if you are running more than one receiver instance behind a load balancer you have added a second source of interleaving on your own side.

The event ledger: one row per event id, forever

Every serious provider puts a stable, unique id on the event itself — evt_... on Stripe, the X-GitHub-Delivery header on GitHub, X-Shopify-Event-Id on Shopify. That id is your idempotency key. Do not invent your own by hashing the body: a provider that re-sends a payload with a refreshed timestamp will hash differently and slip past you.

Store one row per event id in a dedicated table, with the raw payload and a status column. That table is doing three jobs at once — deduplication, an audit log you can hand to a customer arguing about a charge, and the input to the replay tool later on. Keep the payload; the storage is trivial next to the support time it saves. At 50,000 events a month with a 2 KB average payload you are storing 100 MB a month, which costs cents on any managed Postgres.

import asyncio
import os

import asyncpg

DATABASE_URL = os.getenv("DATABASE_URL", "")
if not DATABASE_URL:
    raise RuntimeError("DATABASE_URL is required")

DDL = """
CREATE TABLE IF NOT EXISTS webhook_event (
    event_id     text PRIMARY KEY,
    source       text        NOT NULL,
    event_type   text        NOT NULL,
    payload      jsonb       NOT NULL,
    status       text        NOT NULL DEFAULT 'claimed',
    attempts     integer     NOT NULL DEFAULT 0,
    last_error   text,
    received_at  timestamptz NOT NULL DEFAULT now(),
    settled_at   timestamptz
);

CREATE INDEX IF NOT EXISTS webhook_event_pending_idx
    ON webhook_event (status, received_at)
    WHERE status IN ('claimed', 'failed');
"""


async def migrate() -> None:
    conn = await asyncpg.connect(DATABASE_URL)
    try:
        await conn.execute(DDL)
    finally:
        await conn.close()


if __name__ == "__main__":
    asyncio.run(migrate())

The partial index matters more than it looks. Once the table holds two million settled rows, a worker scanning for pending work against a full index pays for every historical row on every poll. The WHERE clause keeps the index at the size of your actual backlog.

The atomic claim: let the database decide who wins

The naive version is a SELECT to check whether the id exists, then an INSERT if it does not. That is a race, and it fires exactly when you least want it to: two retries of the same event arriving milliseconds apart on two workers both see nothing, both insert, both process. Fix it by making the insert itself the test. INSERT ... ON CONFLICT (event_id) DO NOTHING RETURNING event_id returns a row for the first caller and nothing for everyone after. The primary key constraint is the lock, and it is enforced by Postgres, not by your code.

import os

import asyncpg
import orjson
from fastapi import FastAPI, Request, Response

app = FastAPI()
SOURCE = os.getenv("WEBHOOK_SOURCE", "stripe")
CLAIM_SQL = """
INSERT INTO webhook_event (event_id, source, event_type, payload)
VALUES ($1, $2, $3, $4::jsonb)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id
"""


@app.post("/webhooks/inbound")
async def inbound(request: Request) -> Response:
    raw = await request.body()
    # verify_signature() raises before anything touches the database
    event = verify_signature(raw, dict(request.headers))

    pool: asyncpg.Pool = request.app.state.pool
    async with pool.acquire() as conn:
        claimed = await conn.fetchval(
            CLAIM_SQL,
            event["id"],
            SOURCE,
            event["type"],
            orjson.dumps(event).decode(),
        )

    if claimed is None:
        return Response(status_code=200)  # duplicate, already owned

    await enqueue_job(event["id"])
    return Response(status_code=200)

Two details carry the whole design. First, verification happens before the insert — never write an unverified payload into a table you later trust, and see Verifying Stripe Webhook Signatures for the exact check. Second, a duplicate still returns 200. Returning an error on a duplicate teaches the provider to retry harder, which is the opposite of what you want, and it turns a healthy dedup into a red delivery log the customer sees.

The row's lifecycle, and who is allowed to move it

A claimed row is not a finished row. The worker moves it through processing to settled, or to failed with an incremented attempt counter. Only the transition into processing needs to be exclusive, and UPDATE ... WHERE status = 'claimed' with a RETURNING clause gives you that for free — the update takes a row lock and the second worker sees zero rows affected. If you already run background jobs with Celery, the queue gives you concurrency and the ledger gives you correctness; they are not substitutes.

Versioned state transitions beat "last write wins"

Deduplication solves repeats. It does nothing for order. If subscription.updated v3 lands before v2, both are unique events, both pass the claim, and the later-processed older event overwrites your customer's plan with stale data. The fix is a monotonic version on the entity, not on the event: Stripe objects carry created and Shopify carries updated_at, and both providers include the full object in the payload. Store the version you applied, and make every write conditional on moving forward.

Here is the worker side. Note the match on event type and the single transaction that applies business state, bumps the version, and settles the ledger row together — if anything raises, all three roll back and the retry finds the row exactly as it was.

import os

import asyncpg
import orjson

DATABASE_URL = os.getenv("DATABASE_URL", "")
LEASE_SQL = """
UPDATE webhook_event SET status = 'processing', attempts = attempts + 1
WHERE event_id = $1 AND status IN ('claimed', 'failed')
RETURNING payload
"""
APPLY_SQL = """
UPDATE subscription SET plan = $2, state = $3, version = $4
WHERE stripe_id = $1 AND version < $4
"""


async def process(pool: asyncpg.Pool, event_id: str) -> str:
    async with pool.acquire() as conn, conn.transaction():
        raw = await conn.fetchval(LEASE_SQL, event_id)
        if raw is None:
            return "already-leased"

        event = orjson.loads(raw)
        obj = event["data"]["object"]
        version = int(obj["created"])

        match event["type"]:
            case "customer.subscription.updated" | "customer.subscription.created":
                await conn.execute(
                    APPLY_SQL, obj["id"], obj["items"]["data"][0]["price"]["id"],
                    obj["status"], version,
                )
            case "customer.subscription.deleted":
                await conn.execute(APPLY_SQL, obj["id"], None, "canceled", version)
            case _:
                pass  # unhandled types still settle, so they stop retrying

        await conn.execute(
            "UPDATE webhook_event SET status = 'settled', settled_at = now()"
            " WHERE event_id = $1",
            event_id,
        )
    return "settled"

Where the ledger should live

The Postgres row wins because the dedup marker and the business write commit or roll back together. A Redis marker set before a state write that then fails leaves you permanently deaf to that event — the retry gets skipped as a duplicate and the work never happens. Redis is still useful in front, as a 60-second bloom-style short-circuit that keeps a retry storm off the database, but it is an optimisation, not the source of truth. If your ledger writes start dominating latency, that is a connection-pool problem before it is a storage problem; see Fixing Connection Pool Exhaustion.

When to build this, and when to skip it

Build the full ledger the moment a webhook touches money, provisioning, or anything a customer can see. That is day one for a billing integration. Below roughly 10,000 events a month the whole thing costs a single table and about 30 lines, so the "too early" argument does not survive contact with one duplicated refund.

Skip it when the handler is already naturally idempotent — writing a fact that is true regardless of repetition, like upserting a contact record keyed on email, or setting a flag to the same value. A GitHub webhook that triggers a rebuild of a static site can happily run twice. Also skip the version guard specifically when events carry deltas rather than snapshots: an "increment usage by 5" event has no version to compare, and you must dedupe on event id alone, which is exactly why delta events make the ledger non-optional.

Migrating a receiver that already runs

You can add this to a live endpoint in one deploy without a maintenance window:

1. Ship the webhook_event table and the partial index. Adding an unused table is a zero-risk migration.
2. Insert the claim before your existing handler and log-only for a day: record what would have been skipped, but still process everything. Structured logs make the count easy to pull, as covered in Structured Logging with structlog.
3. Read the duplicate rate. Anything above zero proves the endpoint has been double-processing all along.
4. Flip the claim to enforcing, so claimed is None returns 200 and does no work.
5. Add the version column to the affected business tables, backfill it from the current updated_at, then add AND version < $n to the writes.

Steps 1 to 4 are reversible with an environment flag. Step 5 is the only one that needs care, because a backfill that leaves versions at 0 will let one stale event through per entity.

Replaying the events you missed

Deduplication protects you from too many deliveries. Replay protects you from too few. When your endpoint returns 500 for forty minutes, most providers eventually give up, and those events are gone unless you pull them back. Stripe keeps events queryable for 30 days, GitHub exposes recent deliveries per hook, and both let you page from a timestamp. The replay tool is small because the claim already makes it safe: re-POST everything in the window and let the ledger discard what you already have.

import asyncio
import os

import httpx

API_BASE = os.getenv("STRIPE_API_BASE", "")
API_KEY = os.getenv("STRIPE_API_KEY", "")
RECEIVER_URL = os.getenv("REPLAY_TARGET_URL", "")
REPLAY_TOKEN = os.getenv("REPLAY_TOKEN", "")


async def replay(since_epoch: int) -> dict[str, int]:
    stats = {"fetched": 0, "accepted": 0}
    params = {"created[gte]": since_epoch, "limit": 100}

    async with httpx.AsyncClient(timeout=30.0) as client:
        while True:
            page = await client.get(
                f"{API_BASE}/events", params=params, auth=(API_KEY, "")
            )
            page.raise_for_status()
            body = page.json()

            for event in body["data"]:
                stats["fetched"] += 1
                sent = await client.post(
                    RECEIVER_URL,
                    json=event,
                    headers={"X-Replay-Token": REPLAY_TOKEN},
                )
                if sent.status_code < 300:
                    stats["accepted"] += 1

            if not body.get("has_more"):
                return stats
            params["starting_after"] = body["data"][-1]["id"]


if __name__ == "__main__":
    print(asyncio.run(replay(int(os.getenv("REPLAY_SINCE", "0")))))

Give the replay path its own header token and let the receiver bypass signature verification only for that token, because a re-serialised event no longer matches the original signature. Below is a real run against a 40-minute gap: 4,812 events came back from the provider, 4,790 were already in the ledger, and 22 were genuinely new.

Builder verdict

Put the ledger in Postgres and claim with ON CONFLICT DO NOTHING. It is one table, one insert, and one branch, and it removes an entire category of support ticket that costs you far more than the millisecond it adds. Redis-only dedup looks cheaper on a latency graph and loses events the first time a worker dies mid-transaction; that trade is bad at any volume where webhooks matter. Add the version guard on the same day you add the ledger, because reordering shows up long before you have enough traffic to notice a pattern, and add the replay tool the first time you have an outage — it takes an hour and it is the difference between "we recovered the 22 events" and "please check your account". If you are still deciding whether to accept webhooks at all, the trade-off against pulling on a schedule is laid out in When to Use Webhooks Instead of Polling.

FAQ

What does the ledger cost at a million events a month? About 2 GB of Postgres storage at a 2 KB average payload, which is a few dollars on any managed plan, plus one extra insert per request. Prune settled rows older than 90 days with a scheduled job and the table stabilises. That is far cheaper than one duplicated payout.

Can I dedupe on a hash of the body instead of the event id? Only as a last resort for providers that ship no id. Hashes break when the provider re-serialises the payload or refreshes an embedded timestamp, and then you process the duplicate anyway. Prefer the provider's id, and fall back to a hash of the stable business fields.

Does this survive rotating my webhook signing secret? Yes — the ledger never touches secrets. Verification happens before the claim, so during a rotation you accept both the old and new secret for a few minutes and the dedup logic is unaffected.

How risky is retrofitting this onto a live billing endpoint? Low, if you run the claim in log-only mode for a day first. The insert is additive, the enforcing flip is one environment variable, and the only irreversible step is backfilling the version column on business tables.

Do I still need the ledger if my queue promises exactly-once delivery? Yes. The queue can only deduplicate work it accepted; the duplicate arrives over HTTP before the queue ever sees it. The claim is the boundary between the internet and your system, and no broker can stand in for it.

Related

Same track:

• Processing Webhooks with Python — the receive-verify-enqueue shape this page plugs into.
• Verifying Stripe Webhook Signatures — the check that must run before the claim.
• Handling GitHub Webhooks with FastAPI — a delivery-id source with different retry semantics.

Other tracks:

• Async Database Access with SQLAlchemy — for running the ledger through an ORM instead of raw asyncpg.
• Running Background Jobs with Celery — where the enqueued work actually executes.