[{"data":1,"prerenderedAt":2317},["ShallowReactive",2],{"page-\u002Fautomating-side-hustle-operations-with-apis\u002Fprocessing-webhooks-with-python\u002F":3,"faq-schema-\u002Fautomating-side-hustle-operations-with-apis\u002Fprocessing-webhooks-with-python\u002F":2296},{"id":4,"title":5,"body":6,"description":2286,"extension":2287,"meta":2288,"navigation":389,"path":2292,"seo":2293,"stem":2294,"__hash__":2295},"content\u002Fautomating-side-hustle-operations-with-apis\u002Fprocessing-webhooks-with-python\u002Findex.md","Processing Inbound Webhooks with Python and FastAPI",{"type":7,"value":8,"toc":2270},"minimark",[9,13,42,50,171,174,179,182,192,196,215,265,268,322,326,341,539,565,569,575,963,978,982,989,1310,1339,1343,1361,1451,1454,1532,1546,1550,1562,1708,1729,1733,1740,1760,1764,1866,1876,1880,1928,1932,1935,2118,2145,2149,2171,2175,2181,2187,2201,2207,2213,2217,2222,2244,2249,2266],[10,11,5],"h1",{"id":12},"processing-inbound-webhooks-with-python-and-fastapi",[14,15,16,17,21,22,25,26,31,32,36,37,41],"p",{},"Webhooks are how the rest of the internet tells your automation that something happened — a payment cleared, a repo got pushed, an order shipped. Get them wrong and you either drop events silently or process the same one three times, and both failures cost real money: a missed ",[18,19,20],"code",{},"order.created"," never syncs, a double-processed ",[18,23,24],{},"invoice.paid"," double-charges. This guide builds a production-grade FastAPI receiver that verifies signatures, acknowledges fast, and hands real work to a background worker. It's part of the ",[27,28,30],"a",{"href":29},"\u002Fautomating-side-hustle-operations-with-apis\u002F","Automating Side-Hustle Operations with APIs"," guide, it pairs naturally with ",[27,33,35],{"href":34},"\u002Fautomating-side-hustle-operations-with-apis\u002Fbuilding-zapier-alternatives-with-python\u002F","Building Zapier Alternatives with Python"," — most homegrown automations are just a webhook endpoint plus a queue — and if you are still deciding whether to receive webhooks at all, read ",[27,38,40],{"href":39},"\u002Fgetting-started-with-python-apis-for-builders\u002Funderstanding-rest-vs-graphql\u002Fwhen-to-use-webhooks-instead-of-polling\u002F","When to Use Webhooks Instead of Polling"," first.",[14,43,44,45,49],{},"The trap most builders fall into is treating a webhook like any other POST: parse the JSON, do the work inline, return. That design fails the moment a provider re-sends an event (they all do) or your handler takes longer than the provider's timeout (usually 5–30 seconds). The reliable pattern is narrow and boring: ",[46,47,48],"strong",{},"verify the signature on the raw bytes, store the event id, return 200 immediately, and process asynchronously."," Everything below is that pattern in code, plus the failure modes that only show up once real traffic hits the endpoint.",[51,52,60,61,60,65,60,69,60,88,60,95,60,104,60,112,60,117,60,124,60,129,60,134,60,138,60,142,60,146,60,149,60,153,60,156,60,160,60,164,60,168],"svg",{"viewBox":53,"role":54,"ariaLabelledBy":55,"xmlns":58,"style":59},"0 0 720 240","img",[56,57],"webhooks-flow-t","webhooks-flow-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;height:auto;margin:1.5rem 0;font-family:var(--font-sans);","\n  ",[62,63,64],"title",{"id":56},"Reliable webhook processing flow",[66,67,68],"desc",{"id":57},"A provider posts an event to the FastAPI endpoint, which verifies the HMAC signature, dedupes by event id, returns 200 immediately, and enqueues processing to a background worker.",[70,71,72,73,60],"defs",{},"\n    ",[74,75,82,83,72],"marker",{"id":76,"viewBox":77,"refX":78,"refY":79,"markerWidth":80,"markerHeight":80,"orient":81},"webhooks-arrow","0 0 10 10","9","5","7","auto-start-reverse","\n      ",[84,85],"path",{"d":86,"fill":87},"M0 0 L10 5 L0 10 z","var(--c-text-muted)",[89,90],"rect",{"x":91,"y":91,"width":92,"height":93,"fill":94},"0","720","240","var(--c-surface)",[89,96],{"x":97,"y":98,"width":99,"height":100,"rx":101,"fill":94,"stroke":102,"style":103},"14","90","150","60","10","var(--c-blue)","stroke-width:2;",[105,106,111],"text",{"x":107,"y":108,"fill":109,"style":110},"89","116","var(--c-text)","text-anchor:middle;font-size:13;font-family:var(--font-sans);","Provider",[105,113,116],{"x":107,"y":114,"fill":87,"style":115},"134","text-anchor:middle;font-size:11;font-family:var(--font-sans);","POST event",[118,119],"line",{"x1":120,"y1":121,"x2":122,"y2":121,"stroke":87,"style":123},"164","120","210","stroke-width:2;marker-end:url(#webhooks-arrow);",[89,125],{"x":122,"y":126,"width":127,"height":126,"rx":101,"fill":94,"stroke":128,"style":103},"80","180","var(--c-teal)",[105,130,133],{"x":131,"y":132,"fill":109,"style":110},"300","106","Endpoint",[105,135,137],{"x":131,"y":136,"fill":87,"style":115},"124","verify HMAC",[105,139,141],{"x":131,"y":140,"fill":87,"style":115},"140","dedupe + 200",[118,143],{"x1":144,"y1":121,"x2":145,"y2":121,"stroke":87,"style":123},"390","436",[89,147],{"x":145,"y":98,"width":121,"height":100,"rx":101,"fill":94,"stroke":148,"style":103},"var(--c-yellow)",[105,150,152],{"x":151,"y":108,"fill":109,"style":110},"496","Queue",[105,154,155],{"x":151,"y":114,"fill":87,"style":115},"Redis",[118,157],{"x1":158,"y1":121,"x2":159,"y2":121,"stroke":87,"style":123},"556","602",[89,161],{"x":159,"y":98,"width":162,"height":100,"rx":101,"fill":94,"stroke":163,"style":103},"104","var(--c-coral)",[105,165,167],{"x":166,"y":108,"fill":109,"style":110},"654","Worker",[105,169,170],{"x":166,"y":114,"fill":87,"style":115},"process",[14,172,173],{},"Read that diagram left to right and notice where the boundary sits: the endpoint's only job is the leftmost three boxes — verify, dedupe, acknowledge. The instant it pushes the raw payload onto Redis it is done. Every slow, flaky, or expensive operation lives to the right of the queue, in a worker you can restart, scale, or pause without the provider ever knowing. That split is the entire design. If you internalize nothing else from this guide, internalize the vertical line between \"Endpoint\" and \"Queue.\"",[175,176,178],"h2",{"id":177},"why-fast-acknowledgement-is-a-business-constraint-not-a-nicety","Why fast acknowledgement is a business constraint, not a nicety",[14,180,181],{},"Providers do not wait politely. Stripe gives your endpoint a hard budget in the tens of seconds before it records the delivery as failed; GitHub is tighter; smaller SaaS vendors are often stricter still and undocumented. When you exceed that budget, three things happen at once, and none of them are good. The provider marks the delivery failed even though your code may have succeeded. It schedules a retry, which adds load precisely when you are already slow. And after enough consecutive failures, many providers automatically disable the endpoint entirely, so you stop receiving events until a human logs into a dashboard and re-enables it. A slow handler doesn't just delay one event — it can silence your whole integration.",[14,183,184,185,188,189,191],{},"This is why the acknowledgement is a contract, not a courtesy. The moment you return ",[18,186,187],{},"200",", you are promising the provider \"I have durably taken responsibility for this event.\" You are not promising you have processed it. That distinction is what lets you push the payload to a queue and answer in single-digit milliseconds. It also means the queue must be durable: if you return ",[18,190,187],{}," and then lose the event because it only lived in a Python variable when the process restarted, you have lied to the provider and there is no retry coming, because from their side the delivery succeeded. Redis with append-only persistence, or a real broker, is the price of making that promise honestly.",[175,193,195],{"id":194},"prerequisites","Prerequisites",[14,197,198,199,202,203,206,207,210,211,214],{},"You'll need Python 3.11+ and the following. Install with ",[18,200,201],{},"pip install fastapi \"uvicorn[standard]\" redis",". ",[18,204,205],{},"hmac",", ",[18,208,209],{},"hashlib",", and ",[18,212,213],{},"json"," are standard library.",[216,217,218,235,255],"ul",{},[219,220,221,224,225,229,230,234],"li",{},[46,222,223],{},"FastAPI + Uvicorn"," for the HTTP endpoint. If you're new to it, start with ",[27,226,228],{"href":227},"\u002Fgetting-started-with-python-apis-for-builders\u002Fsetting-up-fastapi\u002F","Setting Up FastAPI"," and pick worker counts using ",[27,231,233],{"href":232},"\u002Fgetting-started-with-python-apis-for-builders\u002Fsetting-up-fastapi\u002Fuvicorn-vs-gunicorn-worker-configuration\u002F","Uvicorn vs Gunicorn worker configuration",".",[219,236,237,240,241,245,246,250,251,234],{},[46,238,239],{},"A queue"," — Redis here, because it doubles as the dedupe store and you may already run it for ",[27,242,244],{"href":243},"\u002Fscaling-and-operating-production-python-apis\u002Fcaching-python-api-responses-with-redis\u002F","caching API responses",". For heavier pipelines with retries, scheduling, and result tracking, graduate to ",[27,247,249],{"href":248},"\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002F","Running Background Jobs with Celery"," — and weigh the alternatives in ",[27,252,254],{"href":253},"\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002Fcelery-vs-rq-vs-arq\u002F","Celery vs RQ vs arq",[219,256,257,260,261,234],{},[46,258,259],{},"Signing secrets in environment variables."," Never hardcode them. Read more on the pattern in ",[27,262,264],{"href":263},"\u002Fgetting-started-with-python-apis-for-builders\u002Fhandling-api-authentication-in-python\u002F","Handling API Authentication in Python",[14,266,267],{},"Set your config in the environment before running anything:",[269,270,275],"pre",{"className":271,"code":272,"language":273,"meta":274,"style":274},"language-bash shiki shiki-themes github-light github-dark","export WEBHOOK_SIGNING_SECRET=\"whsec_replace_me\"\nexport REDIS_URL=\"redis:\u002F\u002Flocalhost:6379\u002F0\"\nexport WEBHOOK_DEDUPE_TTL=\"86400\"\n","bash","",[18,276,277,296,309],{"__ignoreMap":274},[278,279,281,285,289,292],"span",{"class":118,"line":280},1,[278,282,284],{"class":283},"szBVR","export",[278,286,288],{"class":287},"sVt8B"," WEBHOOK_SIGNING_SECRET",[278,290,291],{"class":283},"=",[278,293,295],{"class":294},"sZZnC","\"whsec_replace_me\"\n",[278,297,299,301,304,306],{"class":118,"line":298},2,[278,300,284],{"class":283},[278,302,303],{"class":287}," REDIS_URL",[278,305,291],{"class":283},[278,307,308],{"class":294},"\"redis:\u002F\u002Flocalhost:6379\u002F0\"\n",[278,310,312,314,317,319],{"class":118,"line":311},3,[278,313,284],{"class":283},[278,315,316],{"class":287}," WEBHOOK_DEDUPE_TTL",[278,318,291],{"class":283},[278,320,321],{"class":294},"\"86400\"\n",[175,323,325],{"id":324},"step-1-read-the-raw-body-before-anything-else","Step 1 — Read the raw body before anything else",[14,327,328,329,332,333,336,337,340],{},"The single most common webhook bug is letting FastAPI parse the JSON before you verify the signature. HMAC is computed over the ",[46,330,331],{},"exact bytes"," the provider sent. If you re-serialize a parsed dict, key ordering, unicode escaping, and whitespace all change, and the signature will never match. This is why you must never declare a Pydantic model as the endpoint's body parameter for the verification step — doing so tells FastAPI to consume, parse, and re-encode the stream before your code runs. Always read ",[18,334,335],{},"await request.body()"," first, verify those bytes, and only then call ",[18,338,339],{},"json.loads"," on the same bytes you verified.",[269,342,346],{"className":343,"code":344,"language":345,"meta":274,"style":274},"language-python shiki shiki-themes github-light github-dark","import os\nimport hmac\nimport hashlib\nfrom fastapi import FastAPI, Request, HTTPException\n\napp = FastAPI()\nSIGNING_SECRET = os.getenv(\"WEBHOOK_SIGNING_SECRET\", \"\")\n\n\ndef verify_signature(raw_body: bytes, signature: str) -> bool:\n    \"\"\"Constant-time compare of an HMAC-SHA256 hex digest.\"\"\"\n    if not SIGNING_SECRET:\n        raise RuntimeError(\"WEBHOOK_SIGNING_SECRET is not set\")\n    expected = hmac.new(\n        SIGNING_SECRET.encode(), raw_body, hashlib.sha256\n    ).hexdigest()\n    return hmac.compare_digest(expected, signature)\n","python",[18,347,348,356,363,370,384,391,402,426,431,436,467,473,487,504,515,524,530],{"__ignoreMap":274},[278,349,350,353],{"class":118,"line":280},[278,351,352],{"class":283},"import",[278,354,355],{"class":287}," os\n",[278,357,358,360],{"class":118,"line":298},[278,359,352],{"class":283},[278,361,362],{"class":287}," hmac\n",[278,364,365,367],{"class":118,"line":311},[278,366,352],{"class":283},[278,368,369],{"class":287}," hashlib\n",[278,371,373,376,379,381],{"class":118,"line":372},4,[278,374,375],{"class":283},"from",[278,377,378],{"class":287}," fastapi ",[278,380,352],{"class":283},[278,382,383],{"class":287}," FastAPI, Request, HTTPException\n",[278,385,387],{"class":118,"line":386},5,[278,388,390],{"emptyLinePlaceholder":389},true,"\n",[278,392,394,397,399],{"class":118,"line":393},6,[278,395,396],{"class":287},"app ",[278,398,291],{"class":283},[278,400,401],{"class":287}," FastAPI()\n",[278,403,405,409,412,415,418,420,423],{"class":118,"line":404},7,[278,406,408],{"class":407},"sj4cs","SIGNING_SECRET",[278,410,411],{"class":283}," =",[278,413,414],{"class":287}," os.getenv(",[278,416,417],{"class":294},"\"WEBHOOK_SIGNING_SECRET\"",[278,419,206],{"class":287},[278,421,422],{"class":294},"\"\"",[278,424,425],{"class":287},")\n",[278,427,429],{"class":118,"line":428},8,[278,430,390],{"emptyLinePlaceholder":389},[278,432,434],{"class":118,"line":433},9,[278,435,390],{"emptyLinePlaceholder":389},[278,437,439,442,446,449,452,455,458,461,464],{"class":118,"line":438},10,[278,440,441],{"class":283},"def",[278,443,445],{"class":444},"sScJk"," verify_signature",[278,447,448],{"class":287},"(raw_body: ",[278,450,451],{"class":407},"bytes",[278,453,454],{"class":287},", signature: ",[278,456,457],{"class":407},"str",[278,459,460],{"class":287},") -> ",[278,462,463],{"class":407},"bool",[278,465,466],{"class":287},":\n",[278,468,470],{"class":118,"line":469},11,[278,471,472],{"class":294},"    \"\"\"Constant-time compare of an HMAC-SHA256 hex digest.\"\"\"\n",[278,474,476,479,482,485],{"class":118,"line":475},12,[278,477,478],{"class":283},"    if",[278,480,481],{"class":283}," not",[278,483,484],{"class":407}," SIGNING_SECRET",[278,486,466],{"class":287},[278,488,490,493,496,499,502],{"class":118,"line":489},13,[278,491,492],{"class":283},"        raise",[278,494,495],{"class":407}," RuntimeError",[278,497,498],{"class":287},"(",[278,500,501],{"class":294},"\"WEBHOOK_SIGNING_SECRET is not set\"",[278,503,425],{"class":287},[278,505,507,510,512],{"class":118,"line":506},14,[278,508,509],{"class":287},"    expected ",[278,511,291],{"class":283},[278,513,514],{"class":287}," hmac.new(\n",[278,516,518,521],{"class":118,"line":517},15,[278,519,520],{"class":407},"        SIGNING_SECRET",[278,522,523],{"class":287},".encode(), raw_body, hashlib.sha256\n",[278,525,527],{"class":118,"line":526},16,[278,528,529],{"class":287},"    ).hexdigest()\n",[278,531,533,536],{"class":118,"line":532},17,[278,534,535],{"class":283},"    return",[278,537,538],{"class":287}," hmac.compare_digest(expected, signature)\n",[14,540,541,544,545,548,549,552,553,556,557,560,561,234],{},[18,542,543],{},"hmac.compare_digest"," is non-negotiable — a naive ",[18,546,547],{},"=="," short-circuits on the first differing byte, and that timing difference is measurable across enough requests, letting a patient attacker recover the expected digest one byte at a time. The compare-digest routine runs in time that does not depend on where the mismatch is, which closes that channel. One more subtlety worth budgeting for: providers disagree on how they encode the signature header. Stripe sends a timestamp and a ",[18,550,551],{},"v1="," hex digest joined with commas; GitHub sends ",[18,554,555],{},"sha256="," prefixed hex; others send raw base64. Strip the scheme prefix before comparing, and if you receive multiple candidate signatures in one header (during a secret rotation the provider may sign with both the old and new secret), accept the request if ",[46,558,559],{},"any"," candidate verifies. That is exactly the mechanism that lets you ",[27,562,564],{"href":563},"\u002Fgetting-started-with-python-apis-for-builders\u002Fhandling-api-authentication-in-python\u002Frotating-api-keys-without-downtime\u002F","rotate signing keys without downtime",[175,566,568],{"id":567},"step-2-build-the-endpoint-verify-dedupe-ack-fast","Step 2 — Build the endpoint: verify, dedupe, ack fast",[14,570,571,572,574],{},"The endpoint does four things and nothing else: grab raw bytes, verify, dedupe by event id, enqueue. It returns ",[18,573,187],{}," in single-digit milliseconds because it never touches your business logic.",[269,576,578],{"className":343,"code":577,"language":345,"meta":274,"style":274},"import json\nimport redis.asyncio as redis\n\nr = redis.from_url(os.getenv(\"REDIS_URL\", \"redis:\u002F\u002Flocalhost:6379\u002F0\"))\nDEDUPE_TTL = int(os.getenv(\"WEBHOOK_DEDUPE_TTL\", \"86400\"))\n\n\n@app.post(\"\u002Fwebhooks\u002Finbound\")\nasync def receive_webhook(request: Request):\n    raw = await request.body()\n    signature = request.headers.get(\"X-Signature-256\", \"\")\n\n    if not verify_signature(raw, signature):\n        raise HTTPException(status_code=401, detail=\"bad signature\")\n\n    event = json.loads(raw)\n    event_id = event.get(\"id\")\n    if not event_id:\n        raise HTTPException(status_code=400, detail=\"missing event id\")\n\n    # SET NX returns False if the key already exists -> duplicate delivery.\n    is_new = await r.set(f\"webhook:seen:{event_id}\", \"1\", nx=True, ex=DEDUPE_TTL)\n    if not is_new:\n        return {\"status\": \"duplicate\", \"id\": event_id}\n\n    await r.lpush(\"webhook:queue\", raw)\n    return {\"status\": \"accepted\", \"id\": event_id}\n",[18,579,580,587,600,604,625,648,652,656,668,682,695,714,718,727,755,759,769,784,794,819,824,831,888,898,923,928,943],{"__ignoreMap":274},[278,581,582,584],{"class":118,"line":280},[278,583,352],{"class":283},[278,585,586],{"class":287}," json\n",[278,588,589,591,594,597],{"class":118,"line":298},[278,590,352],{"class":283},[278,592,593],{"class":287}," redis.asyncio ",[278,595,596],{"class":283},"as",[278,598,599],{"class":287}," redis\n",[278,601,602],{"class":118,"line":311},[278,603,390],{"emptyLinePlaceholder":389},[278,605,606,609,611,614,617,619,622],{"class":118,"line":372},[278,607,608],{"class":287},"r ",[278,610,291],{"class":283},[278,612,613],{"class":287}," redis.from_url(os.getenv(",[278,615,616],{"class":294},"\"REDIS_URL\"",[278,618,206],{"class":287},[278,620,621],{"class":294},"\"redis:\u002F\u002Flocalhost:6379\u002F0\"",[278,623,624],{"class":287},"))\n",[278,626,627,630,632,635,638,641,643,646],{"class":118,"line":386},[278,628,629],{"class":407},"DEDUPE_TTL",[278,631,411],{"class":283},[278,633,634],{"class":407}," int",[278,636,637],{"class":287},"(os.getenv(",[278,639,640],{"class":294},"\"WEBHOOK_DEDUPE_TTL\"",[278,642,206],{"class":287},[278,644,645],{"class":294},"\"86400\"",[278,647,624],{"class":287},[278,649,650],{"class":118,"line":393},[278,651,390],{"emptyLinePlaceholder":389},[278,653,654],{"class":118,"line":404},[278,655,390],{"emptyLinePlaceholder":389},[278,657,658,661,663,666],{"class":118,"line":428},[278,659,660],{"class":444},"@app.post",[278,662,498],{"class":287},[278,664,665],{"class":294},"\"\u002Fwebhooks\u002Finbound\"",[278,667,425],{"class":287},[278,669,670,673,676,679],{"class":118,"line":433},[278,671,672],{"class":283},"async",[278,674,675],{"class":283}," def",[278,677,678],{"class":444}," receive_webhook",[278,680,681],{"class":287},"(request: Request):\n",[278,683,684,687,689,692],{"class":118,"line":438},[278,685,686],{"class":287},"    raw ",[278,688,291],{"class":283},[278,690,691],{"class":283}," await",[278,693,694],{"class":287}," request.body()\n",[278,696,697,700,702,705,708,710,712],{"class":118,"line":469},[278,698,699],{"class":287},"    signature ",[278,701,291],{"class":283},[278,703,704],{"class":287}," request.headers.get(",[278,706,707],{"class":294},"\"X-Signature-256\"",[278,709,206],{"class":287},[278,711,422],{"class":294},[278,713,425],{"class":287},[278,715,716],{"class":118,"line":475},[278,717,390],{"emptyLinePlaceholder":389},[278,719,720,722,724],{"class":118,"line":489},[278,721,478],{"class":283},[278,723,481],{"class":283},[278,725,726],{"class":287}," verify_signature(raw, signature):\n",[278,728,729,731,734,738,740,743,745,748,750,753],{"class":118,"line":506},[278,730,492],{"class":283},[278,732,733],{"class":287}," HTTPException(",[278,735,737],{"class":736},"s4XuR","status_code",[278,739,291],{"class":283},[278,741,742],{"class":407},"401",[278,744,206],{"class":287},[278,746,747],{"class":736},"detail",[278,749,291],{"class":283},[278,751,752],{"class":294},"\"bad signature\"",[278,754,425],{"class":287},[278,756,757],{"class":118,"line":517},[278,758,390],{"emptyLinePlaceholder":389},[278,760,761,764,766],{"class":118,"line":526},[278,762,763],{"class":287},"    event ",[278,765,291],{"class":283},[278,767,768],{"class":287}," json.loads(raw)\n",[278,770,771,774,776,779,782],{"class":118,"line":532},[278,772,773],{"class":287},"    event_id ",[278,775,291],{"class":283},[278,777,778],{"class":287}," event.get(",[278,780,781],{"class":294},"\"id\"",[278,783,425],{"class":287},[278,785,787,789,791],{"class":118,"line":786},18,[278,788,478],{"class":283},[278,790,481],{"class":283},[278,792,793],{"class":287}," event_id:\n",[278,795,797,799,801,803,805,808,810,812,814,817],{"class":118,"line":796},19,[278,798,492],{"class":283},[278,800,733],{"class":287},[278,802,737],{"class":736},[278,804,291],{"class":283},[278,806,807],{"class":407},"400",[278,809,206],{"class":287},[278,811,747],{"class":736},[278,813,291],{"class":283},[278,815,816],{"class":294},"\"missing event id\"",[278,818,425],{"class":287},[278,820,822],{"class":118,"line":821},20,[278,823,390],{"emptyLinePlaceholder":389},[278,825,827],{"class":118,"line":826},21,[278,828,830],{"class":829},"sJ8bj","    # SET NX returns False if the key already exists -> duplicate delivery.\n",[278,832,834,837,839,841,844,847,850,853,856,859,862,864,867,869,872,874,877,879,882,884,886],{"class":118,"line":833},22,[278,835,836],{"class":287},"    is_new ",[278,838,291],{"class":283},[278,840,691],{"class":283},[278,842,843],{"class":287}," r.set(",[278,845,846],{"class":283},"f",[278,848,849],{"class":294},"\"webhook:seen:",[278,851,852],{"class":407},"{",[278,854,855],{"class":287},"event_id",[278,857,858],{"class":407},"}",[278,860,861],{"class":294},"\"",[278,863,206],{"class":287},[278,865,866],{"class":294},"\"1\"",[278,868,206],{"class":287},[278,870,871],{"class":736},"nx",[278,873,291],{"class":283},[278,875,876],{"class":407},"True",[278,878,206],{"class":287},[278,880,881],{"class":736},"ex",[278,883,291],{"class":283},[278,885,629],{"class":407},[278,887,425],{"class":287},[278,889,891,893,895],{"class":118,"line":890},23,[278,892,478],{"class":283},[278,894,481],{"class":283},[278,896,897],{"class":287}," is_new:\n",[278,899,901,904,907,910,913,916,918,920],{"class":118,"line":900},24,[278,902,903],{"class":283},"        return",[278,905,906],{"class":287}," {",[278,908,909],{"class":294},"\"status\"",[278,911,912],{"class":287},": ",[278,914,915],{"class":294},"\"duplicate\"",[278,917,206],{"class":287},[278,919,781],{"class":294},[278,921,922],{"class":287},": event_id}\n",[278,924,926],{"class":118,"line":925},25,[278,927,390],{"emptyLinePlaceholder":389},[278,929,931,934,937,940],{"class":118,"line":930},26,[278,932,933],{"class":283},"    await",[278,935,936],{"class":287}," r.lpush(",[278,938,939],{"class":294},"\"webhook:queue\"",[278,941,942],{"class":287},", raw)\n",[278,944,946,948,950,952,954,957,959,961],{"class":118,"line":945},27,[278,947,535],{"class":283},[278,949,906],{"class":287},[278,951,909],{"class":294},[278,953,912],{"class":287},[278,955,956],{"class":294},"\"accepted\"",[278,958,206],{"class":287},[278,960,781],{"class":294},[278,962,922],{"class":287},[14,964,965,966,969,970,973,974,234],{},"The ",[18,967,968],{},"SET ... NX EX"," is an atomic \"claim this id if nobody else has.\" It's your idempotency guard and it survives concurrent deliveries because Redis serializes the operation on a single thread — two simultaneous copies of the same event race for the key, exactly one wins, and the loser gets ",[18,971,972],{},"duplicate",". There is one honest weakness in the naive version above: if the process crashes in the microseconds between claiming the key and pushing to the queue, the event is marked seen but never enqueued, and the retry will be deduped away. In practice that window is vanishingly small, but if you are moving money you should close it by enqueuing first and claiming inside the worker, or by using a Redis transaction that does both atomically. The design trade-off — claim-then-enqueue versus enqueue-then-claim — is worked through in full in ",[27,975,977],{"href":976},"\u002Fautomating-side-hustle-operations-with-apis\u002Fprocessing-webhooks-with-python\u002Fbuilding-an-idempotent-webhook-receiver\u002F","Building an Idempotent Webhook Receiver",[175,979,981],{"id":980},"step-3-drain-the-queue-in-a-background-worker","Step 3 — Drain the queue in a background worker",[14,983,984,985,234],{},"The worker is a separate process so a slow downstream API can never delay an acknowledgement. It pops jobs off the list and runs the real automation — updating a CRM, syncing a spreadsheet, kicking off the kind of cross-platform fan-out described in ",[27,986,988],{"href":987},"\u002Fautomating-side-hustle-operations-with-apis\u002Fconnecting-crm-email-apis\u002F","Connecting CRM & Email APIs",[269,990,992],{"className":343,"code":991,"language":345,"meta":274,"style":274},"import asyncio\nimport json\nimport logging\n\nlogging.basicConfig(level=logging.INFO)\nlog = logging.getLogger(\"webhook-worker\")\n\n\nasync def handle_event(event: dict) -> None:\n    match event.get(\"type\"):\n        case \"order.created\":\n            log.info(\"syncing order %s\", event[\"id\"])\n            # await sync_order_to_sheet(event)\n        case \"customer.updated\":\n            log.info(\"updating CRM record %s\", event[\"id\"])\n        case _:\n            log.info(\"ignoring %s\", event.get(\"type\"))\n\n\nasync def worker() -> None:\n    while True:\n        _, raw = await r.brpop(\"webhook:queue\")\n        try:\n            await handle_event(json.loads(raw))\n        except Exception:\n            log.exception(\"processing failed; sending to dead-letter\")\n            await r.lpush(\"webhook:dead\", raw)\n\n\nif __name__ == \"__main__\":\n    asyncio.run(worker())\n",[18,993,994,1001,1007,1014,1018,1036,1051,1055,1059,1081,1094,1104,1125,1130,1139,1156,1163,1181,1185,1189,1205,1215,1231,1238,1246,1256,1266,1277,1282,1287,1304],{"__ignoreMap":274},[278,995,996,998],{"class":118,"line":280},[278,997,352],{"class":283},[278,999,1000],{"class":287}," asyncio\n",[278,1002,1003,1005],{"class":118,"line":298},[278,1004,352],{"class":283},[278,1006,586],{"class":287},[278,1008,1009,1011],{"class":118,"line":311},[278,1010,352],{"class":283},[278,1012,1013],{"class":287}," logging\n",[278,1015,1016],{"class":118,"line":372},[278,1017,390],{"emptyLinePlaceholder":389},[278,1019,1020,1023,1026,1028,1031,1034],{"class":118,"line":386},[278,1021,1022],{"class":287},"logging.basicConfig(",[278,1024,1025],{"class":736},"level",[278,1027,291],{"class":283},[278,1029,1030],{"class":287},"logging.",[278,1032,1033],{"class":407},"INFO",[278,1035,425],{"class":287},[278,1037,1038,1041,1043,1046,1049],{"class":118,"line":393},[278,1039,1040],{"class":287},"log ",[278,1042,291],{"class":283},[278,1044,1045],{"class":287}," logging.getLogger(",[278,1047,1048],{"class":294},"\"webhook-worker\"",[278,1050,425],{"class":287},[278,1052,1053],{"class":118,"line":404},[278,1054,390],{"emptyLinePlaceholder":389},[278,1056,1057],{"class":118,"line":428},[278,1058,390],{"emptyLinePlaceholder":389},[278,1060,1061,1063,1065,1068,1071,1074,1076,1079],{"class":118,"line":433},[278,1062,672],{"class":283},[278,1064,675],{"class":283},[278,1066,1067],{"class":444}," handle_event",[278,1069,1070],{"class":287},"(event: ",[278,1072,1073],{"class":407},"dict",[278,1075,460],{"class":287},[278,1077,1078],{"class":407},"None",[278,1080,466],{"class":287},[278,1082,1083,1086,1088,1091],{"class":118,"line":438},[278,1084,1085],{"class":283},"    match",[278,1087,778],{"class":287},[278,1089,1090],{"class":294},"\"type\"",[278,1092,1093],{"class":287},"):\n",[278,1095,1096,1099,1102],{"class":118,"line":469},[278,1097,1098],{"class":283},"        case",[278,1100,1101],{"class":294}," \"order.created\"",[278,1103,466],{"class":287},[278,1105,1106,1109,1112,1115,1117,1120,1122],{"class":118,"line":475},[278,1107,1108],{"class":287},"            log.info(",[278,1110,1111],{"class":294},"\"syncing order ",[278,1113,1114],{"class":407},"%s",[278,1116,861],{"class":294},[278,1118,1119],{"class":287},", event[",[278,1121,781],{"class":294},[278,1123,1124],{"class":287},"])\n",[278,1126,1127],{"class":118,"line":489},[278,1128,1129],{"class":829},"            # await sync_order_to_sheet(event)\n",[278,1131,1132,1134,1137],{"class":118,"line":506},[278,1133,1098],{"class":283},[278,1135,1136],{"class":294}," \"customer.updated\"",[278,1138,466],{"class":287},[278,1140,1141,1143,1146,1148,1150,1152,1154],{"class":118,"line":517},[278,1142,1108],{"class":287},[278,1144,1145],{"class":294},"\"updating CRM record ",[278,1147,1114],{"class":407},[278,1149,861],{"class":294},[278,1151,1119],{"class":287},[278,1153,781],{"class":294},[278,1155,1124],{"class":287},[278,1157,1158,1160],{"class":118,"line":526},[278,1159,1098],{"class":283},[278,1161,1162],{"class":287}," _:\n",[278,1164,1165,1167,1170,1172,1174,1177,1179],{"class":118,"line":532},[278,1166,1108],{"class":287},[278,1168,1169],{"class":294},"\"ignoring ",[278,1171,1114],{"class":407},[278,1173,861],{"class":294},[278,1175,1176],{"class":287},", event.get(",[278,1178,1090],{"class":294},[278,1180,624],{"class":287},[278,1182,1183],{"class":118,"line":786},[278,1184,390],{"emptyLinePlaceholder":389},[278,1186,1187],{"class":118,"line":796},[278,1188,390],{"emptyLinePlaceholder":389},[278,1190,1191,1193,1195,1198,1201,1203],{"class":118,"line":821},[278,1192,672],{"class":283},[278,1194,675],{"class":283},[278,1196,1197],{"class":444}," worker",[278,1199,1200],{"class":287},"() -> ",[278,1202,1078],{"class":407},[278,1204,466],{"class":287},[278,1206,1207,1210,1213],{"class":118,"line":826},[278,1208,1209],{"class":283},"    while",[278,1211,1212],{"class":407}," True",[278,1214,466],{"class":287},[278,1216,1217,1220,1222,1224,1227,1229],{"class":118,"line":833},[278,1218,1219],{"class":287},"        _, raw ",[278,1221,291],{"class":283},[278,1223,691],{"class":283},[278,1225,1226],{"class":287}," r.brpop(",[278,1228,939],{"class":294},[278,1230,425],{"class":287},[278,1232,1233,1236],{"class":118,"line":890},[278,1234,1235],{"class":283},"        try",[278,1237,466],{"class":287},[278,1239,1240,1243],{"class":118,"line":900},[278,1241,1242],{"class":283},"            await",[278,1244,1245],{"class":287}," handle_event(json.loads(raw))\n",[278,1247,1248,1251,1254],{"class":118,"line":925},[278,1249,1250],{"class":283},"        except",[278,1252,1253],{"class":407}," Exception",[278,1255,466],{"class":287},[278,1257,1258,1261,1264],{"class":118,"line":930},[278,1259,1260],{"class":287},"            log.exception(",[278,1262,1263],{"class":294},"\"processing failed; sending to dead-letter\"",[278,1265,425],{"class":287},[278,1267,1268,1270,1272,1275],{"class":118,"line":945},[278,1269,1242],{"class":283},[278,1271,936],{"class":287},[278,1273,1274],{"class":294},"\"webhook:dead\"",[278,1276,942],{"class":287},[278,1278,1280],{"class":118,"line":1279},28,[278,1281,390],{"emptyLinePlaceholder":389},[278,1283,1285],{"class":118,"line":1284},29,[278,1286,390],{"emptyLinePlaceholder":389},[278,1288,1290,1293,1296,1299,1302],{"class":118,"line":1289},30,[278,1291,1292],{"class":283},"if",[278,1294,1295],{"class":407}," __name__",[278,1297,1298],{"class":283}," ==",[278,1300,1301],{"class":294}," \"__main__\"",[278,1303,466],{"class":287},[278,1305,1307],{"class":118,"line":1306},31,[278,1308,1309],{"class":287},"    asyncio.run(worker())\n",[14,1311,1312,1313,1316,1317,1320,1321,1325,1326,1329,1330,1334,1335,1338],{},"A failed job lands on a dead-letter list instead of vanishing. You can inspect ",[18,1314,1315],{},"webhook:dead"," and replay it once the downstream bug is fixed. Two production refinements belong here as your volume grows. First, cap retries rather than dead-lettering on the first exception — a ",[18,1318,1319],{},"502"," from a downstream API is usually transient, so retry the same job a bounded number of times with backoff before giving up. The ",[27,1322,1324],{"href":1323},"\u002Fgetting-started-with-python-apis-for-builders\u002Fmaking-http-requests-with-requests-library\u002Fretrying-failed-http-requests-with-tenacity\u002F","tenacity"," library wraps ",[18,1327,1328],{},"handle_event"," in a decorator that does exactly this without cluttering the loop. Second, when downstream calls go over the network, use ",[27,1331,1333],{"href":1332},"\u002Fgetting-started-with-python-apis-for-builders\u002Fmaking-http-requests-with-requests-library\u002Fhttpx-vs-requests-for-async\u002F","httpx"," inside the worker, never the blocking ",[18,1336,1337],{},"requests"," library — a synchronous call inside an async worker blocks the whole event loop and quietly serializes every other job behind it.",[175,1340,1342],{"id":1341},"step-4-handle-provider-retries-and-replays","Step 4 — Handle provider retries and replays",[14,1344,1345,1346,1348,1349,1351,1352,1356,1357,1360],{},"Providers retry on any non-2xx and on timeouts, usually with exponential backoff that can stretch across hours or days. Because Step 2 deduplicates on event id, those retries are harmless — the second delivery returns ",[18,1347,972],{}," and the worker never sees it twice. The only thing you must guarantee is that you return ",[18,1350,187],{}," ",[1353,1354,1355],"em",{},"before"," doing slow work, which Step 2 already does. The timeline below is a typical backoff schedule: your endpoint is briefly down during a deploy, the provider keeps retrying with widening gaps, and the first ",[18,1358,1359],{},"2xx"," it receives ends the sequence.",[51,1362,60,1367,60,1370,60,1373,60,1375,60,1381,60,1385,60,1388,60,1393,60,1396,60,1399,60,1402,60,1405,60,1408,60,1412,60,1415,60,1418,60,1422,60,1425,60,1429,60,1432,60,1435,60,1438,60,1440,60,1446],{"viewBox":1363,"role":54,"ariaLabelledBy":1364,"xmlns":58,"style":59},"0 0 720 210",[1365,1366],"webhooks-retry-t","webhooks-retry-d",[62,1368,1369],{"id":1365},"Provider retry backoff schedule ending on first 2xx",[66,1371,1372],{"id":1366},"A horizontal timeline of five delivery attempts at widening intervals: the first two fail while the endpoint is down, the third succeeds with a 200, and the provider stops retrying after the first success.",[89,1374],{"x":91,"y":91,"width":92,"height":122,"fill":94},[118,1376],{"x1":1377,"y1":1378,"x2":1379,"y2":1378,"stroke":1380,"style":103},"50","110","690","var(--c-border)",[1382,1383],"circle",{"cx":98,"cy":1378,"r":1384,"fill":94,"stroke":163,"style":103},"11",[105,1386,1387],{"x":98,"y":100,"fill":109,"style":115},"t = 0s",[105,1389,1392],{"x":98,"y":1390,"fill":87,"style":1391},"78","text-anchor:middle;font-size:10;font-family:var(--font-sans);","attempt 1",[105,1394,1395],{"x":98,"y":99,"fill":87,"style":115},"503 down",[1382,1397],{"cx":1398,"cy":1378,"r":1384,"fill":94,"stroke":163,"style":103},"230",[105,1400,1401],{"x":1398,"y":100,"fill":109,"style":115},"+30s",[105,1403,1404],{"x":1398,"y":1390,"fill":87,"style":1391},"attempt 2",[105,1406,1407],{"x":1398,"y":99,"fill":87,"style":115},"timeout",[1382,1409],{"cx":807,"cy":1378,"r":1410,"fill":94,"stroke":128,"style":1411},"13","stroke-width:3;",[105,1413,1414],{"x":807,"y":100,"fill":109,"style":115},"+2m",[105,1416,1417],{"x":807,"y":1390,"fill":87,"style":1391},"attempt 3",[105,1419,1421],{"x":807,"y":1420,"fill":128,"style":115},"152","200 accepted",[1382,1423],{"cx":1424,"cy":1378,"r":78,"fill":94,"stroke":1380,"style":103},"540",[105,1426,1428],{"x":1424,"y":1427,"fill":87,"style":115},"66","+10m",[105,1430,1431],{"x":1424,"y":99,"fill":87,"style":115},"not sent",[1382,1433],{"cx":1434,"cy":1378,"r":78,"fill":94,"stroke":1380,"style":103},"660",[105,1436,1437],{"x":1434,"y":1427,"fill":87,"style":115},"+1h",[105,1439,1431],{"x":1434,"y":99,"fill":87,"style":115},[118,1441],{"x1":1442,"y1":1443,"x2":1444,"y2":1443,"stroke":128,"style":1445},"410","95","470","stroke-width:2;stroke-dasharray:4 3;",[105,1447,1450],{"x":1448,"y":1449,"fill":87,"style":115},"580","192","retries stop after first 2xx",[14,1452,1453],{},"For deliberate replays — you re-send an old event to test or backfill, or an attacker captures a valid request off the wire — keep a short timestamp check so a genuine payload can't be resent weeks later and re-executed:",[269,1455,1457],{"className":343,"code":1456,"language":345,"meta":274,"style":274},"import time\n\nREPLAY_TOLERANCE = 300  # seconds\n\n\ndef within_tolerance(event_ts: int) -> bool:\n    return abs(time.time() - event_ts) \u003C= REPLAY_TOLERANCE\n",[18,1458,1459,1466,1470,1483,1487,1491,1510],{"__ignoreMap":274},[278,1460,1461,1463],{"class":118,"line":280},[278,1462,352],{"class":283},[278,1464,1465],{"class":287}," time\n",[278,1467,1468],{"class":118,"line":298},[278,1469,390],{"emptyLinePlaceholder":389},[278,1471,1472,1475,1477,1480],{"class":118,"line":311},[278,1473,1474],{"class":407},"REPLAY_TOLERANCE",[278,1476,411],{"class":283},[278,1478,1479],{"class":407}," 300",[278,1481,1482],{"class":829},"  # seconds\n",[278,1484,1485],{"class":118,"line":372},[278,1486,390],{"emptyLinePlaceholder":389},[278,1488,1489],{"class":118,"line":386},[278,1490,390],{"emptyLinePlaceholder":389},[278,1492,1493,1495,1498,1501,1504,1506,1508],{"class":118,"line":393},[278,1494,441],{"class":283},[278,1496,1497],{"class":444}," within_tolerance",[278,1499,1500],{"class":287},"(event_ts: ",[278,1502,1503],{"class":407},"int",[278,1505,460],{"class":287},[278,1507,463],{"class":407},[278,1509,466],{"class":287},[278,1511,1512,1514,1517,1520,1523,1526,1529],{"class":118,"line":404},[278,1513,535],{"class":283},[278,1515,1516],{"class":407}," abs",[278,1518,1519],{"class":287},"(time.time() ",[278,1521,1522],{"class":283},"-",[278,1524,1525],{"class":287}," event_ts) ",[278,1527,1528],{"class":283},"\u003C=",[278,1530,1531],{"class":407}," REPLAY_TOLERANCE\n",[14,1533,1534,1535,1538,1539,1541,1542,1545],{},"Call ",[18,1536,1537],{},"within_tolerance(event[\"created\"])"," right after signature verification and reject anything outside the window with a ",[18,1540,807],{},". The signature proves the payload is authentic; the timestamp proves it is ",[1353,1543,1544],{},"fresh",". You need both, because a signature alone never expires. Note that this reads the timestamp from inside the signed body (or a signed header), never from an unsigned header an attacker could edit — a freshness check on attacker-controlled data is theater.",[175,1547,1549],{"id":1548},"the-event-lifecycle-where-each-delivery-can-end","The event lifecycle: where each delivery can end",[14,1551,1552,1553,1555,1556,1558,1559,1561],{},"It helps to hold the whole state machine in your head. Every inbound request travels one of a few short paths and terminates in exactly one place. Verification failures die immediately with a ",[18,1554,742],{}," and never touch Redis. Duplicates die at the claim step with a fast ",[18,1557,187],{}," so the provider stops retrying. Only genuinely new, authentic events reach the queue, and from there they either finish cleanly or land in the dead-letter list for a human. Mapping these terminal states explicitly is what stops a subtle class of bug where an event silently disappears with no ",[18,1560,187],{},", no error, and no dead-letter entry.",[51,1563,60,1568,60,1571,60,1574,60,1581,60,1583,60,1587,60,1593,60,1597,60,1600,60,1604,60,1607,60,1609,60,1613,60,1616,60,1619,60,1623,60,1626,60,1628,60,1631,60,1633,60,1639,60,1643,60,1646,60,1650,60,1653,60,1657,60,1659,60,1662,60,1665,60,1669,60,1673,60,1678,60,1682,60,1687,60,1692,60,1694,60,1698,60,1702,60,1704],{"viewBox":1564,"role":54,"ariaLabelledBy":1565,"xmlns":58,"style":59},"0 0 720 300",[1566,1567],"webhooks-lifecycle-t","webhooks-lifecycle-d",[62,1569,1570],{"id":1566},"Webhook event lifecycle state machine",[66,1572,1573],{"id":1567},"A received event flows through verify, claim, enqueue, and worker states along a happy path, branching down to 401 rejected, 200 duplicate, done, and dead-letter terminal states.",[70,1575,72,1576,60],{},[74,1577,82,1579,72],{"id":1578,"viewBox":77,"refX":78,"refY":79,"markerWidth":80,"markerHeight":80,"orient":81},"webhooks-arrow2",[84,1580],{"d":86,"fill":87},[89,1582],{"x":91,"y":91,"width":92,"height":131,"fill":94},[89,1584],{"x":1585,"y":100,"width":1378,"height":1586,"rx":78,"fill":94,"stroke":102,"style":103},"20","52",[105,1588,1592],{"x":1589,"y":1590,"fill":109,"style":1591},"75","82","text-anchor:middle;font-size:12;font-family:var(--font-sans);","Received",[105,1594,1596],{"x":1589,"y":1595,"fill":87,"style":1391},"99","raw bytes",[89,1598],{"x":1599,"y":100,"width":1378,"height":1586,"rx":78,"fill":94,"stroke":128,"style":103},"160",[105,1601,1603],{"x":1602,"y":1590,"fill":109,"style":1591},"215","Verify",[105,1605,1606],{"x":1602,"y":1595,"fill":87,"style":1391},"HMAC",[89,1608],{"x":131,"y":100,"width":1378,"height":1586,"rx":78,"fill":94,"stroke":128,"style":103},[105,1610,1612],{"x":1611,"y":1590,"fill":109,"style":1591},"355","Claim id",[105,1614,1615],{"x":1611,"y":1595,"fill":87,"style":1391},"SET NX",[89,1617],{"x":1618,"y":100,"width":1378,"height":1586,"rx":78,"fill":94,"stroke":148,"style":103},"440",[105,1620,1622],{"x":1621,"y":1590,"fill":109,"style":1591},"495","Enqueue",[105,1624,1625],{"x":1621,"y":1595,"fill":87,"style":1391},"Redis list",[89,1627],{"x":1448,"y":100,"width":1378,"height":1586,"rx":78,"fill":94,"stroke":163,"style":103},[105,1629,167],{"x":1630,"y":1590,"fill":109,"style":1591},"635",[105,1632,170],{"x":1630,"y":1595,"fill":87,"style":1391},[118,1634],{"x1":1635,"y1":1636,"x2":1637,"y2":1636,"stroke":87,"style":1638},"130","86","158","stroke-width:2;marker-end:url(#webhooks-arrow2);",[118,1640],{"x1":1641,"y1":1636,"x2":1642,"y2":1636,"stroke":87,"style":1638},"270","298",[118,1644],{"x1":1442,"y1":1636,"x2":1645,"y2":1636,"stroke":87,"style":1638},"438",[118,1647],{"x1":1648,"y1":1636,"x2":1649,"y2":1636,"stroke":87,"style":1638},"550","578",[89,1651],{"x":1599,"y":122,"width":1378,"height":1652,"rx":78,"fill":94,"stroke":163,"style":103},"46",[105,1654,1656],{"x":1602,"y":1655,"fill":109,"style":1591},"238","401 rejected",[89,1658],{"x":131,"y":122,"width":1378,"height":1652,"rx":78,"fill":94,"stroke":1380,"style":103},[105,1660,1661],{"x":1611,"y":1655,"fill":109,"style":1591},"200 duplicate",[89,1663],{"x":1664,"y":122,"width":98,"height":1652,"rx":78,"fill":94,"stroke":128,"style":103},"500",[105,1666,1668],{"x":1667,"y":1655,"fill":109,"style":1591},"545","Done",[89,1670],{"x":1671,"y":122,"width":1672,"height":1652,"rx":78,"fill":94,"stroke":163,"style":103},"600","100",[105,1674,1677],{"x":1675,"y":1676,"fill":109,"style":115},"650","232","Dead-letter",[105,1679,1681],{"x":1675,"y":1680,"fill":87,"style":1391},"247","replayable",[118,1683],{"x1":1602,"y1":1684,"x2":1602,"y2":1685,"stroke":163,"style":1686},"112","208","stroke-width:2;stroke-dasharray:4 3;marker-end:url(#webhooks-arrow2);",[105,1688,1691],{"x":1689,"y":1599,"fill":87,"style":1690},"223","text-anchor:start;font-size:10;font-family:var(--font-sans);","bad sig",[118,1693],{"x1":1611,"y1":1684,"x2":1611,"y2":1685,"stroke":87,"style":1686},[105,1695,1697],{"x":1696,"y":1599,"fill":87,"style":1690},"363","seen id",[118,1699],{"x1":1700,"y1":1684,"x2":1701,"y2":1685,"stroke":128,"style":1638},"620","560",[118,1703],{"x1":1675,"y1":1684,"x2":1675,"y2":1685,"stroke":163,"style":1686},[105,1705,1707],{"x":1706,"y":1599,"fill":87,"style":1690},"668","raised",[14,1709,1710,1711,1713,1714,1716,1717,1719,1720,1724,1725,234],{},"The commercial payoff of drawing this out is that each terminal state maps to a metric you should be graphing. A rising ",[18,1712,742],{}," rate means a secret is misconfigured or someone is probing you. A rising ",[18,1715,972],{}," rate is usually benign but a sudden spike can mean your ",[18,1718,1359],{}," is slow enough that providers are retrying aggressively. And any growth in the dead-letter list is direct evidence of a downstream bug that is silently costing you fulfilled orders or synced records. Provider-specific quirks — Stripe's signed timestamp scheme, GitHub's delivery id header — layer on top of this same skeleton, covered in ",[27,1721,1723],{"href":1722},"\u002Fautomating-side-hustle-operations-with-apis\u002Fprocessing-webhooks-with-python\u002Fverifying-stripe-webhook-signatures\u002F","Verifying Stripe Webhook Signatures"," and ",[27,1726,1728],{"href":1727},"\u002Fautomating-side-hustle-operations-with-apis\u002Fprocessing-webhooks-with-python\u002Fhandling-github-webhooks-with-fastapi\u002F","Handling GitHub Webhooks with FastAPI",[175,1730,1732],{"id":1731},"scaling-the-receiver-and-seeing-inside-it","Scaling the receiver and seeing inside it",[14,1734,1735,1736,1739],{},"The receiver scales almost embarrassingly well because it does so little. A single Uvicorn worker verifying an HMAC and doing one Redis round-trip comfortably handles a few thousand requests per second, and because the work per request is bounded and uniform, latency stays flat as volume climbs — there is no slow query or external call hiding in the hot path to blow out your p99. When you do need more headroom, add Uvicorn workers up to your vCPU count, not beyond; webhook receiving is CPU-light and network-bound, so oversubscribing workers buys you nothing but context-switch overhead. The worker side scales independently: if fulfillment falls behind, run more worker processes draining the same Redis list, and because ",[18,1737,1738],{},"BRPOP"," hands each job to exactly one consumer, they parallelize without you writing any coordination code.",[14,1741,1742,1743,1745,1746,1750,1751,1755,1756,1759],{},"What you cannot skip is observability. A webhook endpoint fails silently by nature — the provider sees a ",[18,1744,187],{},", you see nothing, and the missing side effect only surfaces when a customer complains. Emit a structured log line at every terminal state (accepted, duplicate, rejected, dead-lettered) with the event id and type as fields, then alert on the ratios. Ship those logs the way ",[27,1747,1749],{"href":1748},"\u002Fscaling-and-operating-production-python-apis\u002Fmonitoring-and-logging-python-apis\u002F","Monitoring and Logging Python APIs"," lays out, and prefer machine-parseable output using ",[27,1752,1754],{"href":1753},"\u002Fscaling-and-operating-production-python-apis\u002Fmonitoring-and-logging-python-apis\u002Fstructured-logging-with-structlog\u002F","structured logging with structlog"," so \"how many deliveries did we reject in the last hour\" is a query, not a grep. The single most valuable dashboard panel is queue depth: if ",[18,1757,1758],{},"LLEN webhook:queue"," is climbing, your workers are losing the race against inbound volume and you need capacity before the backlog turns into stale data.",[175,1761,1763],{"id":1762},"configuration-reference","Configuration reference",[1765,1766,1767,1783],"table",{},[1768,1769,1770],"thead",{},[1771,1772,1773,1777,1780],"tr",{},[1774,1775,1776],"th",{},"Env var",[1774,1778,1779],{},"Default",[1774,1781,1782],{},"Production recommendation",[1784,1785,1786,1802,1817,1833,1847],"tbody",{},[1771,1787,1788,1794,1799],{},[1789,1790,1791],"td",{},[18,1792,1793],{},"WEBHOOK_SIGNING_SECRET",[1789,1795,1796],{},[1353,1797,1798],{},"(none — required)",[1789,1800,1801],{},"Unique per provider; rotate quarterly; store in a secret manager",[1771,1803,1804,1809,1814],{},[1789,1805,1806],{},[18,1807,1808],{},"REDIS_URL",[1789,1810,1811],{},[18,1812,1813],{},"redis:\u002F\u002Flocalhost:6379\u002F0",[1789,1815,1816],{},"Managed Redis with TLS, a password, and append-only persistence",[1771,1818,1819,1824,1830],{},[1789,1820,1821],{},[18,1822,1823],{},"WEBHOOK_DEDUPE_TTL",[1789,1825,1826,1829],{},[18,1827,1828],{},"86400"," (24h)",[1789,1831,1832],{},"Match or exceed the provider's max retry window",[1771,1834,1835,1839,1844],{},[1789,1836,1837],{},[18,1838,1474],{},[1789,1840,1841,1843],{},[18,1842,131],{}," (5 min)",[1789,1845,1846],{},"300s is the common default; tighten if clocks are reliable",[1771,1848,1849,1854,1859],{},[1789,1850,1851],{},[18,1852,1853],{},"WEBHOOK_QUEUE_KEY",[1789,1855,1856],{},[18,1857,1858],{},"webhook:queue",[1789,1860,1861,1862,1865],{},"Namespace per environment (",[18,1863,1864],{},"prod:webhook:queue",")",[14,1867,1868,1869,1871,1872,1875],{},"The one setting builders get wrong most often is ",[18,1870,1823],{},". It is tempting to keep it short to save memory, but the value has a correctness meaning: it is the window during which you promise to recognize a repeat. Set it below the provider's maximum retry span and a late retry — the kind that arrives after a multi-hour outage on their side — sails past your dedupe check and gets processed a second time. Memory is cheap; a ",[18,1873,1874],{},"seen"," key is a few dozen bytes, so a million events a day at a 24-hour TTL is single-digit megabytes. Err long.",[175,1877,1879],{"id":1878},"gotchas-failure-modes","Gotchas & failure modes",[1881,1882,1883,1892,1898,1904,1910,1919],"ol",{},[219,1884,1885,1888,1889,1891],{},[46,1886,1887],{},"Parsing the body before verifying."," FastAPI's automatic Pydantic parsing consumes and re-encodes the body. Read ",[18,1890,335],{}," yourself and verify the bytes, then parse.",[219,1893,1894,1897],{},[46,1895,1896],{},"Slow inline handlers."," If your handler calls three external APIs before returning, you'll blow the provider's timeout, get a non-2xx logged on their side, and trigger a retry storm — or an auto-disabled endpoint. Always enqueue and return.",[219,1899,1900,1903],{},[46,1901,1902],{},"No idempotency."," Without the dedupe step, every provider retry re-runs your logic — double-charged customers, double-posted content, duplicated CRM rows. Dedupe on the provider's event id, not on your own request hash.",[219,1905,1906,1909],{},[46,1907,1908],{},"Replay attacks."," A captured-but-valid payload can be resent forever if you only check the signature. Pair signature verification with a timestamp tolerance read from the signed body.",[219,1911,1912,1915,1916,1918],{},[46,1913,1914],{},"Returning 200 on a bad signature."," Some teams swallow errors to \"stop the retries.\" Don't — a ",[18,1917,742],{}," on an unverifiable request is correct, and a verified request should never fail at the ack stage.",[219,1920,1921,1924,1925,1927],{},[46,1922,1923],{},"Losing events on restart."," An in-memory queue drops everything the moment the process dies, after you already promised the provider a ",[18,1926,187],{},". Use a durable store with persistence enabled, and treat \"acknowledged\" as \"durably enqueued,\" never \"held in a variable.\"",[175,1929,1931],{"id":1930},"verification","Verification",[14,1933,1934],{},"Compute a signature locally and POST to the running endpoint. This mirrors exactly what a provider does:",[269,1936,1938],{"className":271,"code":1937,"language":273,"meta":274,"style":274},"BODY='{\"id\":\"evt_123\",\"type\":\"order.created\",\"created\":1750000000}'\nSIG=$(printf '%s' \"$BODY\" | openssl dgst -sha256 -hmac \"$WEBHOOK_SIGNING_SECRET\" | awk '{print $2}')\n\ncurl -s -X POST http:\u002F\u002Flocalhost:8000\u002Fwebhooks\u002Finbound \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"X-Signature-256: $SIG\" \\\n  -d \"$BODY\"\n# {\"status\":\"accepted\",\"id\":\"evt_123\"}\n\n# Send the same request again -> deduped:\ncurl -s -X POST http:\u002F\u002Flocalhost:8000\u002Fwebhooks\u002Finbound \\\n  -H \"X-Signature-256: $SIG\" -d \"$BODY\"\n# {\"status\":\"duplicate\",\"id\":\"evt_123\"}\n",[18,1939,1940,1950,2006,2010,2030,2040,2054,2066,2071,2075,2080,2094,2113],{"__ignoreMap":274},[278,1941,1942,1945,1947],{"class":118,"line":280},[278,1943,1944],{"class":287},"BODY",[278,1946,291],{"class":283},[278,1948,1949],{"class":294},"'{\"id\":\"evt_123\",\"type\":\"order.created\",\"created\":1750000000}'\n",[278,1951,1952,1955,1957,1960,1963,1966,1969,1972,1974,1977,1980,1983,1986,1989,1991,1994,1996,1998,2001,2004],{"class":118,"line":298},[278,1953,1954],{"class":287},"SIG",[278,1956,291],{"class":283},[278,1958,1959],{"class":287},"$(",[278,1961,1962],{"class":407},"printf",[278,1964,1965],{"class":294}," '%s'",[278,1967,1968],{"class":294}," \"",[278,1970,1971],{"class":287},"$BODY",[278,1973,861],{"class":294},[278,1975,1976],{"class":283}," |",[278,1978,1979],{"class":444}," openssl",[278,1981,1982],{"class":294}," dgst",[278,1984,1985],{"class":407}," -sha256",[278,1987,1988],{"class":407}," -hmac",[278,1990,1968],{"class":294},[278,1992,1993],{"class":287},"$WEBHOOK_SIGNING_SECRET",[278,1995,861],{"class":294},[278,1997,1976],{"class":283},[278,1999,2000],{"class":444}," awk",[278,2002,2003],{"class":294}," '{print $2}'",[278,2005,425],{"class":287},[278,2007,2008],{"class":118,"line":311},[278,2009,390],{"emptyLinePlaceholder":389},[278,2011,2012,2015,2018,2021,2024,2027],{"class":118,"line":372},[278,2013,2014],{"class":444},"curl",[278,2016,2017],{"class":407}," -s",[278,2019,2020],{"class":407}," -X",[278,2022,2023],{"class":294}," POST",[278,2025,2026],{"class":294}," http:\u002F\u002Flocalhost:8000\u002Fwebhooks\u002Finbound",[278,2028,2029],{"class":407}," \\\n",[278,2031,2032,2035,2038],{"class":118,"line":386},[278,2033,2034],{"class":407},"  -H",[278,2036,2037],{"class":294}," \"Content-Type: application\u002Fjson\"",[278,2039,2029],{"class":407},[278,2041,2042,2044,2047,2050,2052],{"class":118,"line":393},[278,2043,2034],{"class":407},[278,2045,2046],{"class":294}," \"X-Signature-256: ",[278,2048,2049],{"class":287},"$SIG",[278,2051,861],{"class":294},[278,2053,2029],{"class":407},[278,2055,2056,2059,2061,2063],{"class":118,"line":404},[278,2057,2058],{"class":407},"  -d",[278,2060,1968],{"class":294},[278,2062,1971],{"class":287},[278,2064,2065],{"class":294},"\"\n",[278,2067,2068],{"class":118,"line":428},[278,2069,2070],{"class":829},"# {\"status\":\"accepted\",\"id\":\"evt_123\"}\n",[278,2072,2073],{"class":118,"line":433},[278,2074,390],{"emptyLinePlaceholder":389},[278,2076,2077],{"class":118,"line":438},[278,2078,2079],{"class":829},"# Send the same request again -> deduped:\n",[278,2081,2082,2084,2086,2088,2090,2092],{"class":118,"line":469},[278,2083,2014],{"class":444},[278,2085,2017],{"class":407},[278,2087,2020],{"class":407},[278,2089,2023],{"class":294},[278,2091,2026],{"class":294},[278,2093,2029],{"class":407},[278,2095,2096,2098,2100,2102,2104,2107,2109,2111],{"class":118,"line":475},[278,2097,2034],{"class":407},[278,2099,2046],{"class":294},[278,2101,2049],{"class":287},[278,2103,861],{"class":294},[278,2105,2106],{"class":407}," -d",[278,2108,1968],{"class":294},[278,2110,1971],{"class":287},[278,2112,2065],{"class":294},[278,2114,2115],{"class":118,"line":489},[278,2116,2117],{"class":829},"# {\"status\":\"duplicate\",\"id\":\"evt_123\"}\n",[14,2119,2120,2121,2124,2125,2128,2129,2131,2132,206,2135,210,2137,2139,2140,2144],{},"The worker log should show ",[18,2122,2123],{},"syncing order evt_123"," exactly once. Send a request with a tampered body and you'll get ",[18,2126,2127],{},"401 bad signature",". Before you ship, promote this from a manual ",[18,2130,2014],{}," into an automated test — a small pytest that posts a correctly signed payload, a duplicate, and a tampered one, asserting ",[18,2133,2134],{},"accepted",[18,2136,972],{},[18,2138,742],{}," respectively. The patterns for that live in ",[27,2141,2143],{"href":2142},"\u002Fscaling-and-operating-production-python-apis\u002Ftesting-python-apis-with-pytest\u002F","Testing Python APIs with pytest",", and it is the cheapest insurance you will ever buy against a signature-handling regression that silently rejects real revenue events.",[175,2146,2148],{"id":2147},"cost-performance-note","Cost & performance note",[14,2150,2151,2152,2155,2156,2159,2160,2163,2164,2166,2167,234],{},"A verify-and-enqueue endpoint does microseconds of CPU work plus one Redis round-trip, so a single ",[18,2153,2154],{},"$5","–",[18,2157,2158],{},"$7"," VPS handles thousands of webhooks per second and a month of realistic side-hustle volume — say a million events — costs a rounding error in compute. Concretely: at a million events a month you are looking at two to four dollars of compute if you keep the worker count matched to vCPUs, plus a few megabytes of Redis for dedupe keys. Your real cost lives in the worker, where the outbound API calls happen — meter those the way the ",[27,2161,2162],{"href":29},"parent automation guide"," describes, because a single fulfillment step that calls three paid APIs dwarfs the entire cost of receiving. Returning ",[18,2165,187],{}," in under 50ms also keeps you well inside every provider's timeout, which eliminates the retry traffic that would otherwise multiply your worker load two- or three-fold during a slow patch. And if you later monetize the automation, the same fast-ack pattern feeds straight into billing events — see how to wire that up in ",[27,2168,2170],{"href":2169},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fintegrating-stripe-with-python-apis\u002F","Integrating Stripe with Python APIs",[175,2172,2174],{"id":2173},"faq","FAQ",[14,2176,2177,2180],{},[46,2178,2179],{},"How much does this cost to run at a million webhooks a month?","\nTwo to four dollars of compute if you keep the worker count matched to vCPUs, plus a few megabytes of Redis for dedupe keys — the receiver itself is nearly free. Your spend concentrates in the worker's outbound API calls, so the honest budgeting move is to price one fulfillment (how many paid API calls each event triggers) and multiply by volume, not to worry about the endpoint.",[14,2182,2183,2186],{},[46,2184,2185],{},"Why return 200 before processing instead of after?","\nProviders enforce a delivery timeout in the tens of seconds. If your handler does real work inline and exceeds that, the provider records a failure, retries, and after enough failures may auto-disable your endpoint — even though your code actually succeeded. Acknowledging immediately and processing in a worker decouples your business logic from the provider's clock and protects the whole integration.",[14,2188,2189,2192,2193,2196,2197,2200],{},[46,2190,2191],{},"What should I dedupe on, and what happens if I rotate secrets?","\nDedupe on the provider's own event id (Stripe's ",[18,2194,2195],{},"evt_...",", GitHub's ",[18,2198,2199],{},"X-GitHub-Delivery",") — it is stable across retries of the same event. Secret rotation is independent of dedupe: during a rotation, accept a request if it verifies against either the old or new secret, which lets you roll keys with zero dropped deliveries. Hashing the body works for dedupe too but breaks if the provider re-serializes between retries.",[14,2202,2203,2206],{},[46,2204,2205],{},"Do I really need a queue for a small side hustle?","\nIf your handler is genuinely instant — write one row and return — you can skip it and save yourself a moving part. The moment a handler makes an outbound API call, add the queue, because that is the call that will eventually be slow or flaky and blow your acknowledgement budget. Migrating from inline to queued later is cheap; recovering from a retry storm in production is not.",[14,2208,2209,2212],{},[46,2210,2211],{},"What happens to events while my worker is down, and is there migration risk?","\nThey stay in the durable Redis list until the worker returns and drains them, so the endpoint keeps accepting and acknowledging even when downstream processing is offline. The one migration risk to plan for is changing your event schema: an old payload sitting in the queue or dead-letter list may not match a new handler, so version your handlers or drain the queue before deploying a breaking change.",[175,2214,2216],{"id":2215},"related","Related",[14,2218,2219],{},[46,2220,2221],{},"Same track:",[216,2223,2224,2229,2234,2239],{},[219,2225,2226,2228],{},[27,2227,1723],{"href":1722}," — the provider-specific signing scheme, timestamps, and rolling secrets.",[219,2230,2231,2233],{},[27,2232,1728],{"href":1727}," — the same pattern applied to GitHub's delivery headers and events.",[219,2235,2236,2238],{},[27,2237,977],{"href":976}," — closing the claim-then-enqueue race for money-moving events.",[219,2240,2241,2243],{},[27,2242,35],{"href":34}," — where the webhook endpoint becomes the trigger for a whole automation graph.",[14,2245,2246],{},[46,2247,2248],{},"Other tracks:",[216,2250,2251,2256,2261],{},[219,2252,2253,2255],{},[27,2254,249],{"href":248}," — graduating the worker from a raw Redis loop to a real task system.",[219,2257,2258,2260],{},[27,2259,1749],{"href":1748}," — the dashboards and alerts that stop silent webhook failures.",[219,2262,2263,2265],{},[27,2264,2170],{"href":2169}," — turning inbound events into billing when you monetize the automation.",[2267,2268,2269],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}",{"title":274,"searchDepth":298,"depth":298,"links":2271},[2272,2273,2274,2275,2276,2277,2278,2279,2280,2281,2282,2283,2284,2285],{"id":177,"depth":298,"text":178},{"id":194,"depth":298,"text":195},{"id":324,"depth":298,"text":325},{"id":567,"depth":298,"text":568},{"id":980,"depth":298,"text":981},{"id":1341,"depth":298,"text":1342},{"id":1548,"depth":298,"text":1549},{"id":1731,"depth":298,"text":1732},{"id":1762,"depth":298,"text":1763},{"id":1878,"depth":298,"text":1879},{"id":1930,"depth":298,"text":1931},{"id":2147,"depth":298,"text":2148},{"id":2173,"depth":298,"text":2174},{"id":2215,"depth":298,"text":2216},"Build a reliable FastAPI webhook endpoint: verify HMAC signatures, return 200 fast, enqueue work, dedupe by event id, and survive provider retries.","md",{"pageTitle":2289,"datePublished":2290,"dateModified":2291},"Processing Webhooks with Python and FastAPI","2026-06-18","2026-07-23","\u002Fautomating-side-hustle-operations-with-apis\u002Fprocessing-webhooks-with-python",{"title":5,"description":2286},"automating-side-hustle-operations-with-apis\u002Fprocessing-webhooks-with-python\u002Findex","z-tEJUbeyFepsKiaJZlmX7zl3XS9h1s2aw7sD4YU4as",{"@context":2297,"@type":2298,"mainEntity":2299},"https:\u002F\u002Fschema.org","FAQPage",[2300,2305,2308,2311,2314],{"@type":2301,"name":2179,"acceptedAnswer":2302},"Question",{"@type":2303,"text":2304},"Answer","Two to four dollars of compute if you keep the worker count matched to vCPUs, plus a few megabytes of Redis for dedupe keys — the receiver itself is nearly free. Your spend concentrates in the worker's outbound API calls, so the honest budgeting move is to price one fulfillment (how many paid API calls each event triggers) and multiply by volume, not to worry about the endpoint.",{"@type":2301,"name":2185,"acceptedAnswer":2306},{"@type":2303,"text":2307},"Providers enforce a delivery timeout in the tens of seconds. If your handler does real work inline and exceeds that, the provider records a failure, retries, and after enough failures may auto-disable your endpoint — even though your code actually succeeded. Acknowledging immediately and processing in a worker decouples your business logic from the provider's clock and protects the whole integration.",{"@type":2301,"name":2191,"acceptedAnswer":2309},{"@type":2303,"text":2310},"Dedupe on the provider's own event id (Stripe's evt_..., GitHub's X-GitHub-Delivery) — it is stable across retries of the same event. Secret rotation is independent of dedupe: during a rotation, accept a request if it verifies against either the old or new secret, which lets you roll keys with zero dropped deliveries. Hashing the body works for dedupe too but breaks if the provider re-serializes between retries.",{"@type":2301,"name":2205,"acceptedAnswer":2312},{"@type":2303,"text":2313},"If your handler is genuinely instant — write one row and return — you can skip it and save yourself a moving part. The moment a handler makes an outbound API call, add the queue, because that is the call that will eventually be slow or flaky and blow your acknowledgement budget. Migrating from inline to queued later is cheap; recovering from a retry storm in production is not.",{"@type":2301,"name":2211,"acceptedAnswer":2315},{"@type":2303,"text":2316},"They stay in the durable Redis list until the worker returns and drains them, so the endpoint keeps accepting and acknowledging even when downstream processing is offline. The one migration risk to plan for is changing your event schema: an old payload sitting in the queue or dead-letter list may not match a new handler, so version your handlers or drain the queue before deploying a breaking change.",1784887027767]