[{"data":1,"prerenderedAt":4992},["ShallowReactive",2],{"page-\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002F":3,"faq-schema-\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002F":4971},{"id":4,"title":5,"body":6,"description":4961,"extension":4962,"meta":4963,"navigation":387,"path":4967,"seo":4968,"stem":4969,"__hash__":4970},"content\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Findex.md","Automating AI Workflows with Python APIs",{"type":7,"value":8,"toc":4946},"minimark",[9,13,23,26,175,178,183,203,239,247,250,322,330,332,336,339,347,589,592,594,598,601,609,1283,1297,1405,1407,1411,1414,1425,1959,1968,1976,1978,1982,1985,1988,2401,2412,2525,2527,2531,2538,2544,2643,2954,2961,3038,3040,3044,3047,3057,3060,3580,3592,3595,4051,4063,4065,4069,4236,4245,4247,4251,4272,4293,4301,4315,4338,4340,4344,4347,4350,4473,4488,4495,4503,4683,4709,4711,4715,4741,4841,4843,4847,4853,4864,4873,4889,4903,4907,4942],[10,11,5],"h1",{"id":12},"automating-ai-workflows-with-python-apis",[14,15,16,17,22],"p",{},"The moment a language model stops being a chat window and starts being a step inside a background job, everything you already know about calling third-party APIs applies again — and a few things that do not apply anywhere else start to bite. Part of the ",[18,19,21],"a",{"href":20},"\u002Fautomating-side-hustle-operations-with-apis\u002F","Automating Side-Hustle Operations with APIs"," guide. This page builds one production-grade AI pipeline: a worker that pulls jobs off a queue, calls a model with a versioned prompt, forces structured output, validates it, retries the failures that deserve retrying, and refuses to spend more than the budget you gave it.",[14,24,25],{},"The framing that matters is commercial. A model call is the single most expensive request in your stack — often two to four orders of magnitude more expensive than a database query — and it is also the only one whose cost varies with the length of the input a customer happened to paste in. Treat it like an unmetered dependency and your margin evaporates on the day you get traction. Every pattern below exists to make that cost predictable.",[27,28,36,37,36,41,36,45,36,52,36,71,36,79,36,88,36,94,36,101,36,105,36,109,36,113,36,117,36,121,36,125,36,129,36,133,36,139,36,142,36,146,36,151,36,155,36,158,36,160,36,163,36,165,36,167,36,170],"svg",{"viewBox":29,"role":30,"ariaLabelledBy":31,"xmlns":34,"style":35},"0 0 760 290","img",[32,33],"aiwf-pipe-t","aiwf-pipe-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;height:auto;margin:1.5rem 0;font-family:var(--font-sans);","\n  ",[38,39,40],"title",{"id":32},"Background AI job pipeline from queue to stored result",[42,43,44],"desc",{"id":33},"Jobs flow from a Redis queue into an async worker pool, which loads a versioned prompt, calls the model API, records spend in a cost ledger, validates the response against a schema, and writes typed rows to a results store.",[46,47],"rect",{"x":48,"y":48,"width":49,"height":50,"fill":51},"0","760","290","var(--c-surface)",[53,54,55,56,36],"defs",{},"\n    ",[57,58,65,66,55],"marker",{"id":59,"viewBox":60,"refX":61,"refY":62,"markerWidth":63,"markerHeight":63,"orient":64},"aiwf-pipe-arrow","0 0 10 10","9","5","7","auto-start-reverse","\n      ",[67,68],"path",{"d":69,"fill":70},"M0 0 L10 5 L0 10 z","var(--c-text-muted)",[72,73,78],"text",{"x":74,"y":75,"fill":76,"style":77},"380","28","var(--c-text)","text-anchor:middle;font-size:14;font-family:var(--font-sans);","One AI job: queue in, typed row out",[46,80],{"x":81,"y":82,"width":83,"height":84,"rx":85,"fill":51,"stroke":86,"style":87},"20","64","150","60","10","var(--c-blue)","stroke-width:2;",[72,89,93],{"x":90,"y":91,"fill":76,"style":92},"95","100","text-anchor:middle;font-size:13;font-family:var(--font-sans);","Job queue",[95,96],"line",{"x1":97,"y1":98,"x2":99,"y2":98,"stroke":70,"style":100},"170","94","200","stroke-width:2;marker-end:url(#aiwf-pipe-arrow);",[46,102],{"x":103,"y":82,"width":83,"height":84,"rx":85,"fill":51,"stroke":104,"style":87},"205","var(--c-teal)",[72,106,108],{"x":107,"y":91,"fill":76,"style":92},"280","Async workers",[95,110],{"x1":111,"y1":98,"x2":112,"y2":98,"stroke":70,"style":100},"355","385",[46,114],{"x":115,"y":82,"width":83,"height":84,"rx":85,"fill":51,"stroke":116,"style":87},"390","var(--c-coral)",[72,118,120],{"x":119,"y":91,"fill":76,"style":92},"465","Model API call",[95,122],{"x1":123,"y1":98,"x2":124,"y2":98,"stroke":70,"style":100},"540","570",[46,126],{"x":127,"y":82,"width":128,"height":84,"rx":85,"fill":51,"stroke":86,"style":87},"575","165",[72,130,132],{"x":131,"y":91,"fill":76,"style":92},"657","Schema validator",[72,134,138],{"x":135,"y":136,"fill":70,"style":137},"185","86","text-anchor:middle;font-size:11;font-family:var(--font-sans);","N",[72,140,72],{"x":141,"y":136,"fill":70,"style":137},"557",[95,143],{"x1":107,"y1":144,"x2":107,"y2":145,"stroke":70,"style":100},"186","130",[46,147],{"x":103,"y":144,"width":83,"height":148,"rx":85,"fill":149,"stroke":150,"style":87},"56","var(--c-surface-alt)","var(--c-border)",[72,152,154],{"x":107,"y":153,"fill":76,"style":92},"220","Prompt registry",[95,156],{"x1":119,"y1":145,"x2":119,"y2":157,"stroke":70,"style":100},"182",[46,159],{"x":115,"y":144,"width":83,"height":148,"rx":85,"fill":149,"stroke":150,"style":87},[72,161,162],{"x":119,"y":153,"fill":76,"style":92},"Cost ledger",[95,164],{"x1":131,"y1":145,"x2":131,"y2":157,"stroke":70,"style":100},[46,166],{"x":127,"y":144,"width":128,"height":148,"rx":85,"fill":149,"stroke":150,"style":87},[72,168,169],{"x":131,"y":153,"fill":76,"style":92},"Results store",[72,171,174],{"x":74,"y":172,"fill":70,"style":173},"270","text-anchor:middle;font-size:12;font-family:var(--font-sans);","Every arrow is a place a job can fail — and every failure has a different retry rule",[176,177],"hr",{},[179,180,182],"h2",{"id":181},"prerequisites","Prerequisites",[14,184,185,186,190,191,194,195,198,199,202],{},"You need Python 3.11 or newer. The code below uses ",[187,188,189],"code",{},"match"," statements, ",[187,192,193],{},"tomllib",", and the ",[187,196,197],{},"X | None"," union syntax without a ",[187,200,201],{},"__future__"," import, all of which land in 3.11. Install the model SDK, a validation library, a retry library, and a Redis client:",[204,205,210],"pre",{"className":206,"code":207,"language":208,"meta":209,"style":209},"language-bash shiki shiki-themes github-light github-dark","pip install \"anthropic>=0.68\" \"pydantic>=2.7\" \"tenacity>=8.3\" \"redis>=5\" \"structlog>=24.1\"\n","bash","",[187,211,212],{"__ignoreMap":209},[213,214,216,220,224,227,230,233,236],"span",{"class":95,"line":215},1,[213,217,219],{"class":218},"sScJk","pip",[213,221,223],{"class":222},"sZZnC"," install",[213,225,226],{"class":222}," \"anthropic>=0.68\"",[213,228,229],{"class":222}," \"pydantic>=2.7\"",[213,231,232],{"class":222}," \"tenacity>=8.3\"",[213,234,235],{"class":222}," \"redis>=5\"",[213,237,238],{"class":222}," \"structlog>=24.1\"\n",[14,240,241,242,246],{},"The Anthropic SDK ships its own async HTTP transport built on httpx, so you do not add httpx yourself for model calls. You still want httpx directly for the non-model hops in the same pipeline — pulling the source document, posting the result to a webhook — and the ",[18,243,245],{"href":244},"\u002Fgetting-started-with-python-apis-for-builders\u002Fmaking-http-requests-with-requests-library\u002Fhttpx-vs-requests-for-async\u002F","httpx versus requests trade-offs for async work"," apply to those hops unchanged.",[14,248,249],{},"Set configuration in the environment. Never hardcode a model name, a concurrency number, or a spend cap:",[204,251,255],{"className":252,"code":253,"language":254,"meta":209,"style":209},"language-env shiki shiki-themes github-light github-dark","ANTHROPIC_API_KEY=sk-ant-...\nLLM_MODEL=claude-opus-4-8\nLLM_CHEAP_MODEL=claude-haiku-4-5\nLLM_MAX_TOKENS=2048\nLLM_REQUEST_TIMEOUT_S=90\nLLM_CONCURRENCY=8\nLLM_MAX_ATTEMPTS=4\nPROMPT_FILE=prompts\u002Flead_extract.toml\nPROMPT_VERSION=v3\nDAILY_BUDGET_USD=25.00\nREDIS_URL=redis:\u002F\u002Flocalhost:6379\u002F0\n","env",[187,256,257,262,268,274,280,286,292,298,304,310,316],{"__ignoreMap":209},[213,258,259],{"class":95,"line":215},[213,260,261],{},"ANTHROPIC_API_KEY=sk-ant-...\n",[213,263,265],{"class":95,"line":264},2,[213,266,267],{},"LLM_MODEL=claude-opus-4-8\n",[213,269,271],{"class":95,"line":270},3,[213,272,273],{},"LLM_CHEAP_MODEL=claude-haiku-4-5\n",[213,275,277],{"class":95,"line":276},4,[213,278,279],{},"LLM_MAX_TOKENS=2048\n",[213,281,283],{"class":95,"line":282},5,[213,284,285],{},"LLM_REQUEST_TIMEOUT_S=90\n",[213,287,289],{"class":95,"line":288},6,[213,290,291],{},"LLM_CONCURRENCY=8\n",[213,293,295],{"class":95,"line":294},7,[213,296,297],{},"LLM_MAX_ATTEMPTS=4\n",[213,299,301],{"class":95,"line":300},8,[213,302,303],{},"PROMPT_FILE=prompts\u002Flead_extract.toml\n",[213,305,307],{"class":95,"line":306},9,[213,308,309],{},"PROMPT_VERSION=v3\n",[213,311,313],{"class":95,"line":312},10,[213,314,315],{},"DAILY_BUDGET_USD=25.00\n",[213,317,319],{"class":95,"line":318},11,[213,320,321],{},"REDIS_URL=redis:\u002F\u002Flocalhost:6379\u002F0\n",[14,323,324,325,329],{},"The baseline assumption is that this worker runs outside your web process. If you are calling a model inside a request handler, you are holding an HTTP connection open for the length of a generation, which is the wrong shape for anything but streaming — see ",[18,326,328],{"href":327},"\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fstreaming-llm-responses-through-fastapi\u002F","streaming LLM responses through FastAPI"," for that case. Everything here targets background work, which is where the volume, and therefore the money, actually is.",[176,331],{},[179,333,335],{"id":334},"step-1-build-one-configured-client-not-a-client-per-call","Step 1: Build one configured client, not a client per call",[14,337,338],{},"The first mistake in almost every AI pipeline is constructing the SDK client inside the function that uses it. Each construction spins up a fresh connection pool, so a batch of fifty jobs opens fifty TLS handshakes to the same host and throws them away. Build one async client at import time, cache it, and give it an explicit timeout.",[14,340,341,342,346],{},"Turn the SDK's built-in retries ",[343,344,345],"strong",{},"off",". The SDK retries 429s and 5xx errors twice by default with its own backoff, which silently multiplies your wall-clock timeout by three and hides the failure from your metrics. You want retries in one place, under your control, with your logging attached.",[204,348,352],{"className":349,"code":350,"language":351,"meta":209,"style":209},"language-python shiki shiki-themes github-light github-dark","# llm\u002Fclient.py\nimport os\nfrom functools import lru_cache\n\nfrom anthropic import AsyncAnthropic\n\nMODEL = os.getenv(\"LLM_MODEL\", \"claude-opus-4-8\")\nCHEAP_MODEL = os.getenv(\"LLM_CHEAP_MODEL\", \"claude-haiku-4-5\")\nMAX_TOKENS = int(os.getenv(\"LLM_MAX_TOKENS\", \"2048\"))\nREQUEST_TIMEOUT_S = float(os.getenv(\"LLM_REQUEST_TIMEOUT_S\", \"90\"))\n\n\n@lru_cache(maxsize=1)\ndef get_client() -> AsyncAnthropic:\n    \"\"\"One pooled async client per process. API key comes from the environment.\"\"\"\n    return AsyncAnthropic(\n        timeout=REQUEST_TIMEOUT_S,   # seconds in the Python SDK\n        max_retries=0,               # we own the retry policy, see Step 4\n    )\n","python",[187,353,354,360,370,383,389,401,405,429,448,472,494,498,503,524,536,542,551,567,583],{"__ignoreMap":209},[213,355,356],{"class":95,"line":215},[213,357,359],{"class":358},"sJ8bj","# llm\u002Fclient.py\n",[213,361,362,366],{"class":95,"line":264},[213,363,365],{"class":364},"szBVR","import",[213,367,369],{"class":368},"sVt8B"," os\n",[213,371,372,375,378,380],{"class":95,"line":270},[213,373,374],{"class":364},"from",[213,376,377],{"class":368}," functools ",[213,379,365],{"class":364},[213,381,382],{"class":368}," lru_cache\n",[213,384,385],{"class":95,"line":276},[213,386,388],{"emptyLinePlaceholder":387},true,"\n",[213,390,391,393,396,398],{"class":95,"line":282},[213,392,374],{"class":364},[213,394,395],{"class":368}," anthropic ",[213,397,365],{"class":364},[213,399,400],{"class":368}," AsyncAnthropic\n",[213,402,403],{"class":95,"line":288},[213,404,388],{"emptyLinePlaceholder":387},[213,406,407,411,414,417,420,423,426],{"class":95,"line":294},[213,408,410],{"class":409},"sj4cs","MODEL",[213,412,413],{"class":364}," =",[213,415,416],{"class":368}," os.getenv(",[213,418,419],{"class":222},"\"LLM_MODEL\"",[213,421,422],{"class":368},", ",[213,424,425],{"class":222},"\"claude-opus-4-8\"",[213,427,428],{"class":368},")\n",[213,430,431,434,436,438,441,443,446],{"class":95,"line":300},[213,432,433],{"class":409},"CHEAP_MODEL",[213,435,413],{"class":364},[213,437,416],{"class":368},[213,439,440],{"class":222},"\"LLM_CHEAP_MODEL\"",[213,442,422],{"class":368},[213,444,445],{"class":222},"\"claude-haiku-4-5\"",[213,447,428],{"class":368},[213,449,450,453,455,458,461,464,466,469],{"class":95,"line":306},[213,451,452],{"class":409},"MAX_TOKENS",[213,454,413],{"class":364},[213,456,457],{"class":409}," int",[213,459,460],{"class":368},"(os.getenv(",[213,462,463],{"class":222},"\"LLM_MAX_TOKENS\"",[213,465,422],{"class":368},[213,467,468],{"class":222},"\"2048\"",[213,470,471],{"class":368},"))\n",[213,473,474,477,479,482,484,487,489,492],{"class":95,"line":312},[213,475,476],{"class":409},"REQUEST_TIMEOUT_S",[213,478,413],{"class":364},[213,480,481],{"class":409}," float",[213,483,460],{"class":368},[213,485,486],{"class":222},"\"LLM_REQUEST_TIMEOUT_S\"",[213,488,422],{"class":368},[213,490,491],{"class":222},"\"90\"",[213,493,471],{"class":368},[213,495,496],{"class":95,"line":318},[213,497,388],{"emptyLinePlaceholder":387},[213,499,501],{"class":95,"line":500},12,[213,502,388],{"emptyLinePlaceholder":387},[213,504,506,509,512,516,519,522],{"class":95,"line":505},13,[213,507,508],{"class":218},"@lru_cache",[213,510,511],{"class":368},"(",[213,513,515],{"class":514},"s4XuR","maxsize",[213,517,518],{"class":364},"=",[213,520,521],{"class":409},"1",[213,523,428],{"class":368},[213,525,527,530,533],{"class":95,"line":526},14,[213,528,529],{"class":364},"def",[213,531,532],{"class":218}," get_client",[213,534,535],{"class":368},"() -> AsyncAnthropic:\n",[213,537,539],{"class":95,"line":538},15,[213,540,541],{"class":222},"    \"\"\"One pooled async client per process. API key comes from the environment.\"\"\"\n",[213,543,545,548],{"class":95,"line":544},16,[213,546,547],{"class":364},"    return",[213,549,550],{"class":368}," AsyncAnthropic(\n",[213,552,554,557,559,561,564],{"class":95,"line":553},17,[213,555,556],{"class":514},"        timeout",[213,558,518],{"class":364},[213,560,476],{"class":409},[213,562,563],{"class":368},",   ",[213,565,566],{"class":358},"# seconds in the Python SDK\n",[213,568,570,573,575,577,580],{"class":95,"line":569},18,[213,571,572],{"class":514},"        max_retries",[213,574,518],{"class":364},[213,576,48],{"class":409},[213,578,579],{"class":368},",               ",[213,581,582],{"class":358},"# we own the retry policy, see Step 4\n",[213,584,586],{"class":95,"line":585},19,[213,587,588],{"class":368},"    )\n",[14,590,591],{},"Ninety seconds sounds enormous next to the two-second timeout you would give a REST call. It is correct here. A model generating two thousand tokens of structured output genuinely takes twenty to sixty seconds under load, and a timeout tuned for a normal API produces a pipeline that fails constantly while the model is doing exactly what you asked. Set the timeout from measured p99 latency, not from habit.",[176,593],{},[179,595,597],{"id":596},"step-2-force-structured-output-with-a-schema-never-a-regex","Step 2: Force structured output with a schema, never a regex",[14,599,600],{},"The single highest-leverage change you can make to an AI pipeline is to stop parsing prose. Ask for JSON in the prompt and roughly 2% of responses will arrive wrapped in a markdown fence, prefixed with \"Here is the JSON you requested\", or truncated mid-object. At a thousand jobs a day that is twenty broken rows every day, forever.",[14,602,603,604,608],{},"Constrain the response format at the API level instead. Define the shape once as a Pydantic model, hand the generated JSON Schema to the request, and validate what comes back anyway — belt and braces, because the constraint guarantees shape, not truth. If Pydantic v2 is new to you, ",[18,605,607],{"href":606},"\u002Fgetting-started-with-python-apis-for-builders\u002Fparsing-json-responses\u002Fvalidating-json-with-pydantic-v2\u002F","validating JSON with Pydantic v2"," covers the model syntax in depth.",[204,610,612],{"className":349,"code":611,"language":351,"meta":209,"style":209},"# llm\u002Fextract.py\nimport os\nfrom typing import Literal\n\nfrom pydantic import BaseModel, ValidationError\n\nfrom .client import MODEL, MAX_TOKENS, get_client\nfrom .prompts import load_prompt\n\n\nclass Lead(BaseModel):\n    company: str\n    contact_email: str\n    budget_usd: int | None = None\n    intent: Literal[\"demo\", \"pricing\", \"support\", \"other\"]\n\n\nSCHEMA = Lead.model_json_schema()\nSCHEMA[\"additionalProperties\"] = False   # required by the structured-output API\n\n\nclass ExtractionRefused(Exception):\n    \"\"\"Permanent: the model declined. Retrying burns money and never succeeds.\"\"\"\n\n\nclass ExtractionTruncated(Exception):\n    \"\"\"Recoverable by raising max_tokens, not by retrying identically.\"\"\"\n\n\nasync def extract_lead(raw: str) -> tuple[Lead, dict]:\n    system_text, prompt_digest = load_prompt()\n    resp = await get_client().messages.create(\n        model=MODEL,\n        max_tokens=MAX_TOKENS,\n        system=[{\n            \"type\": \"text\",\n            \"text\": system_text,\n            \"cache_control\": {\"type\": \"ephemeral\"},   # cache the stable prefix\n        }],\n        output_config={\"format\": {\"type\": \"json_schema\", \"schema\": SCHEMA}},\n        messages=[{\"role\": \"user\", \"content\": raw}],\n    )\n\n    match resp.stop_reason:\n        case \"refusal\":\n            raise ExtractionRefused(str(getattr(resp, \"stop_details\", None)))\n        case \"max_tokens\":\n            raise ExtractionTruncated(f\"hit max_tokens={MAX_TOKENS}\")\n\n    text = \"\".join(b.text for b in resp.content if b.type == \"text\")\n    meta = {\n        \"prompt_digest\": prompt_digest,\n        \"input_tokens\": resp.usage.input_tokens,\n        \"output_tokens\": resp.usage.output_tokens,\n        \"cache_read_tokens\": resp.usage.cache_read_input_tokens,\n    }\n    return Lead.model_validate_json(text), meta\n",[187,613,614,619,625,637,641,653,657,676,688,692,696,712,720,727,746,772,776,780,790,811,816,821,836,842,847,852,866,872,877,882,909,920,934,947,959,970,984,993,1016,1022,1057,1084,1089,1094,1103,1115,1145,1155,1177,1182,1222,1233,1242,1251,1260,1269,1275],{"__ignoreMap":209},[213,615,616],{"class":95,"line":215},[213,617,618],{"class":358},"# llm\u002Fextract.py\n",[213,620,621,623],{"class":95,"line":264},[213,622,365],{"class":364},[213,624,369],{"class":368},[213,626,627,629,632,634],{"class":95,"line":270},[213,628,374],{"class":364},[213,630,631],{"class":368}," typing ",[213,633,365],{"class":364},[213,635,636],{"class":368}," Literal\n",[213,638,639],{"class":95,"line":276},[213,640,388],{"emptyLinePlaceholder":387},[213,642,643,645,648,650],{"class":95,"line":282},[213,644,374],{"class":364},[213,646,647],{"class":368}," pydantic ",[213,649,365],{"class":364},[213,651,652],{"class":368}," BaseModel, ValidationError\n",[213,654,655],{"class":95,"line":288},[213,656,388],{"emptyLinePlaceholder":387},[213,658,659,661,664,666,669,671,673],{"class":95,"line":294},[213,660,374],{"class":364},[213,662,663],{"class":368}," .client ",[213,665,365],{"class":364},[213,667,668],{"class":409}," MODEL",[213,670,422],{"class":368},[213,672,452],{"class":409},[213,674,675],{"class":368},", get_client\n",[213,677,678,680,683,685],{"class":95,"line":300},[213,679,374],{"class":364},[213,681,682],{"class":368}," .prompts ",[213,684,365],{"class":364},[213,686,687],{"class":368}," load_prompt\n",[213,689,690],{"class":95,"line":306},[213,691,388],{"emptyLinePlaceholder":387},[213,693,694],{"class":95,"line":312},[213,695,388],{"emptyLinePlaceholder":387},[213,697,698,701,704,706,709],{"class":95,"line":318},[213,699,700],{"class":364},"class",[213,702,703],{"class":218}," Lead",[213,705,511],{"class":368},[213,707,708],{"class":218},"BaseModel",[213,710,711],{"class":368},"):\n",[213,713,714,717],{"class":95,"line":500},[213,715,716],{"class":368},"    company: ",[213,718,719],{"class":409},"str\n",[213,721,722,725],{"class":95,"line":505},[213,723,724],{"class":368},"    contact_email: ",[213,726,719],{"class":409},[213,728,729,732,735,738,741,743],{"class":95,"line":526},[213,730,731],{"class":368},"    budget_usd: ",[213,733,734],{"class":409},"int",[213,736,737],{"class":364}," |",[213,739,740],{"class":409}," None",[213,742,413],{"class":364},[213,744,745],{"class":409}," None\n",[213,747,748,751,754,756,759,761,764,766,769],{"class":95,"line":538},[213,749,750],{"class":368},"    intent: Literal[",[213,752,753],{"class":222},"\"demo\"",[213,755,422],{"class":368},[213,757,758],{"class":222},"\"pricing\"",[213,760,422],{"class":368},[213,762,763],{"class":222},"\"support\"",[213,765,422],{"class":368},[213,767,768],{"class":222},"\"other\"",[213,770,771],{"class":368},"]\n",[213,773,774],{"class":95,"line":544},[213,775,388],{"emptyLinePlaceholder":387},[213,777,778],{"class":95,"line":553},[213,779,388],{"emptyLinePlaceholder":387},[213,781,782,785,787],{"class":95,"line":569},[213,783,784],{"class":409},"SCHEMA",[213,786,413],{"class":364},[213,788,789],{"class":368}," Lead.model_json_schema()\n",[213,791,792,794,797,800,803,805,808],{"class":95,"line":585},[213,793,784],{"class":409},[213,795,796],{"class":368},"[",[213,798,799],{"class":222},"\"additionalProperties\"",[213,801,802],{"class":368},"] ",[213,804,518],{"class":364},[213,806,807],{"class":409}," False",[213,809,810],{"class":358},"   # required by the structured-output API\n",[213,812,814],{"class":95,"line":813},20,[213,815,388],{"emptyLinePlaceholder":387},[213,817,819],{"class":95,"line":818},21,[213,820,388],{"emptyLinePlaceholder":387},[213,822,824,826,829,831,834],{"class":95,"line":823},22,[213,825,700],{"class":364},[213,827,828],{"class":218}," ExtractionRefused",[213,830,511],{"class":368},[213,832,833],{"class":409},"Exception",[213,835,711],{"class":368},[213,837,839],{"class":95,"line":838},23,[213,840,841],{"class":222},"    \"\"\"Permanent: the model declined. Retrying burns money and never succeeds.\"\"\"\n",[213,843,845],{"class":95,"line":844},24,[213,846,388],{"emptyLinePlaceholder":387},[213,848,850],{"class":95,"line":849},25,[213,851,388],{"emptyLinePlaceholder":387},[213,853,855,857,860,862,864],{"class":95,"line":854},26,[213,856,700],{"class":364},[213,858,859],{"class":218}," ExtractionTruncated",[213,861,511],{"class":368},[213,863,833],{"class":409},[213,865,711],{"class":368},[213,867,869],{"class":95,"line":868},27,[213,870,871],{"class":222},"    \"\"\"Recoverable by raising max_tokens, not by retrying identically.\"\"\"\n",[213,873,875],{"class":95,"line":874},28,[213,876,388],{"emptyLinePlaceholder":387},[213,878,880],{"class":95,"line":879},29,[213,881,388],{"emptyLinePlaceholder":387},[213,883,885,888,891,894,897,900,903,906],{"class":95,"line":884},30,[213,886,887],{"class":364},"async",[213,889,890],{"class":364}," def",[213,892,893],{"class":218}," extract_lead",[213,895,896],{"class":368},"(raw: ",[213,898,899],{"class":409},"str",[213,901,902],{"class":368},") -> tuple[Lead, ",[213,904,905],{"class":409},"dict",[213,907,908],{"class":368},"]:\n",[213,910,912,915,917],{"class":95,"line":911},31,[213,913,914],{"class":368},"    system_text, prompt_digest ",[213,916,518],{"class":364},[213,918,919],{"class":368}," load_prompt()\n",[213,921,923,926,928,931],{"class":95,"line":922},32,[213,924,925],{"class":368},"    resp ",[213,927,518],{"class":364},[213,929,930],{"class":364}," await",[213,932,933],{"class":368}," get_client().messages.create(\n",[213,935,937,940,942,944],{"class":95,"line":936},33,[213,938,939],{"class":514},"        model",[213,941,518],{"class":364},[213,943,410],{"class":409},[213,945,946],{"class":368},",\n",[213,948,950,953,955,957],{"class":95,"line":949},34,[213,951,952],{"class":514},"        max_tokens",[213,954,518],{"class":364},[213,956,452],{"class":409},[213,958,946],{"class":368},[213,960,962,965,967],{"class":95,"line":961},35,[213,963,964],{"class":514},"        system",[213,966,518],{"class":364},[213,968,969],{"class":368},"[{\n",[213,971,973,976,979,982],{"class":95,"line":972},36,[213,974,975],{"class":222},"            \"type\"",[213,977,978],{"class":368},": ",[213,980,981],{"class":222},"\"text\"",[213,983,946],{"class":368},[213,985,987,990],{"class":95,"line":986},37,[213,988,989],{"class":222},"            \"text\"",[213,991,992],{"class":368},": system_text,\n",[213,994,996,999,1002,1005,1007,1010,1013],{"class":95,"line":995},38,[213,997,998],{"class":222},"            \"cache_control\"",[213,1000,1001],{"class":368},": {",[213,1003,1004],{"class":222},"\"type\"",[213,1006,978],{"class":368},[213,1008,1009],{"class":222},"\"ephemeral\"",[213,1011,1012],{"class":368},"},   ",[213,1014,1015],{"class":358},"# cache the stable prefix\n",[213,1017,1019],{"class":95,"line":1018},39,[213,1020,1021],{"class":368},"        }],\n",[213,1023,1025,1028,1030,1033,1036,1038,1040,1042,1045,1047,1050,1052,1054],{"class":95,"line":1024},40,[213,1026,1027],{"class":514},"        output_config",[213,1029,518],{"class":364},[213,1031,1032],{"class":368},"{",[213,1034,1035],{"class":222},"\"format\"",[213,1037,1001],{"class":368},[213,1039,1004],{"class":222},[213,1041,978],{"class":368},[213,1043,1044],{"class":222},"\"json_schema\"",[213,1046,422],{"class":368},[213,1048,1049],{"class":222},"\"schema\"",[213,1051,978],{"class":368},[213,1053,784],{"class":409},[213,1055,1056],{"class":368},"}},\n",[213,1058,1060,1063,1065,1068,1071,1073,1076,1078,1081],{"class":95,"line":1059},41,[213,1061,1062],{"class":514},"        messages",[213,1064,518],{"class":364},[213,1066,1067],{"class":368},"[{",[213,1069,1070],{"class":222},"\"role\"",[213,1072,978],{"class":368},[213,1074,1075],{"class":222},"\"user\"",[213,1077,422],{"class":368},[213,1079,1080],{"class":222},"\"content\"",[213,1082,1083],{"class":368},": raw}],\n",[213,1085,1087],{"class":95,"line":1086},42,[213,1088,588],{"class":368},[213,1090,1092],{"class":95,"line":1091},43,[213,1093,388],{"emptyLinePlaceholder":387},[213,1095,1097,1100],{"class":95,"line":1096},44,[213,1098,1099],{"class":364},"    match",[213,1101,1102],{"class":368}," resp.stop_reason:\n",[213,1104,1106,1109,1112],{"class":95,"line":1105},45,[213,1107,1108],{"class":364},"        case",[213,1110,1111],{"class":222}," \"refusal\"",[213,1113,1114],{"class":368},":\n",[213,1116,1118,1121,1124,1126,1128,1131,1134,1137,1139,1142],{"class":95,"line":1117},46,[213,1119,1120],{"class":364},"            raise",[213,1122,1123],{"class":368}," ExtractionRefused(",[213,1125,899],{"class":409},[213,1127,511],{"class":368},[213,1129,1130],{"class":409},"getattr",[213,1132,1133],{"class":368},"(resp, ",[213,1135,1136],{"class":222},"\"stop_details\"",[213,1138,422],{"class":368},[213,1140,1141],{"class":409},"None",[213,1143,1144],{"class":368},")))\n",[213,1146,1148,1150,1153],{"class":95,"line":1147},47,[213,1149,1108],{"class":364},[213,1151,1152],{"class":222}," \"max_tokens\"",[213,1154,1114],{"class":368},[213,1156,1158,1160,1163,1166,1169,1172,1175],{"class":95,"line":1157},48,[213,1159,1120],{"class":364},[213,1161,1162],{"class":368}," ExtractionTruncated(",[213,1164,1165],{"class":364},"f",[213,1167,1168],{"class":222},"\"hit max_tokens=",[213,1170,1171],{"class":409},"{MAX_TOKENS}",[213,1173,1174],{"class":222},"\"",[213,1176,428],{"class":368},[213,1178,1180],{"class":95,"line":1179},49,[213,1181,388],{"emptyLinePlaceholder":387},[213,1183,1185,1188,1190,1193,1196,1199,1202,1205,1208,1211,1214,1217,1220],{"class":95,"line":1184},50,[213,1186,1187],{"class":368},"    text ",[213,1189,518],{"class":364},[213,1191,1192],{"class":222}," \"\"",[213,1194,1195],{"class":368},".join(b.text ",[213,1197,1198],{"class":364},"for",[213,1200,1201],{"class":368}," b ",[213,1203,1204],{"class":364},"in",[213,1206,1207],{"class":368}," resp.content ",[213,1209,1210],{"class":364},"if",[213,1212,1213],{"class":368}," b.type ",[213,1215,1216],{"class":364},"==",[213,1218,1219],{"class":222}," \"text\"",[213,1221,428],{"class":368},[213,1223,1225,1228,1230],{"class":95,"line":1224},51,[213,1226,1227],{"class":368},"    meta ",[213,1229,518],{"class":364},[213,1231,1232],{"class":368}," {\n",[213,1234,1236,1239],{"class":95,"line":1235},52,[213,1237,1238],{"class":222},"        \"prompt_digest\"",[213,1240,1241],{"class":368},": prompt_digest,\n",[213,1243,1245,1248],{"class":95,"line":1244},53,[213,1246,1247],{"class":222},"        \"input_tokens\"",[213,1249,1250],{"class":368},": resp.usage.input_tokens,\n",[213,1252,1254,1257],{"class":95,"line":1253},54,[213,1255,1256],{"class":222},"        \"output_tokens\"",[213,1258,1259],{"class":368},": resp.usage.output_tokens,\n",[213,1261,1263,1266],{"class":95,"line":1262},55,[213,1264,1265],{"class":222},"        \"cache_read_tokens\"",[213,1267,1268],{"class":368},": resp.usage.cache_read_input_tokens,\n",[213,1270,1272],{"class":95,"line":1271},56,[213,1273,1274],{"class":368},"    }\n",[213,1276,1278,1280],{"class":95,"line":1277},57,[213,1279,547],{"class":364},[213,1281,1282],{"class":368}," Lead.model_validate_json(text), meta\n",[14,1284,1285,1286,1288,1289,1292,1293,1296],{},"Two details earn their keep. The ",[187,1287,189],{}," on ",[187,1290,1291],{},"stop_reason"," separates the three ways a call can end badly, and each one wants a different response: a refusal is permanent, a truncation is a configuration problem, and everything else is a transient you can retry. And ",[187,1294,1295],{},"cache_control"," on the system block tells the API to cache that prefix, which is where most of the savings in Step 6 come from.",[27,1298,36,1303,36,1306,36,1309,36,1313,36,1320,36,1325,36,1330,36,1334,36,1339,36,1343,36,1347,36,1350,36,1353,36,1357,36,1361,36,1363,36,1366,36,1370,36,1373,36,1376,36,1378,36,1380,36,1383,36,1387,36,1391,36,1395,36,1400],{"viewBox":1299,"role":30,"ariaLabelledBy":1300,"xmlns":34,"style":35},"0 0 720 285",[1301,1302],"aiwf-extract-t","aiwf-extract-d",[38,1304,1305],{"id":1301},"Path a raw document takes through schema-constrained extraction",[42,1307,1308],{"id":1302},"Raw text goes into a schema-constrained model call, the response text is validated by Pydantic, valid rows are persisted, and invalid rows are repaired once before landing in a dead-letter queue.",[46,1310],{"x":48,"y":48,"width":1311,"height":1312,"fill":51},"720","285",[53,1314,55,1315,36],{},[57,1316,65,1318,55],{"id":1317,"viewBox":60,"refX":61,"refY":62,"markerWidth":63,"markerHeight":63,"orient":64},"aiwf-extract-arrow",[67,1319],{"d":69,"fill":70},[72,1321,1324],{"x":1322,"y":1323,"fill":76,"style":77},"360","26","Shape is guaranteed by the API; truth is guaranteed by you",[46,1326],{"x":81,"y":1327,"width":1328,"height":1329,"rx":61,"fill":51,"stroke":150,"style":87},"48","160","54",[72,1331,1333],{"x":91,"y":1332,"fill":76,"style":173},"80","Raw text",[95,1335],{"x1":1336,"y1":1337,"x2":103,"y2":1337,"stroke":70,"style":1338},"180","75","stroke-width:2;marker-end:url(#aiwf-extract-arrow);",[46,1340],{"x":1341,"y":1327,"width":1342,"height":1329,"rx":61,"fill":51,"stroke":86,"style":87},"210","230",[72,1344,1346],{"x":1345,"y":1332,"fill":76,"style":173},"325","Schema-constrained call",[95,1348],{"x1":1349,"y1":1337,"x2":119,"y2":1337,"stroke":70,"style":1338},"440",[46,1351],{"x":1352,"y":1327,"width":153,"height":1329,"rx":61,"fill":51,"stroke":86,"style":87},"470",[72,1354,1356],{"x":1355,"y":1332,"fill":76,"style":173},"580","Response text",[95,1358],{"x1":1355,"y1":1359,"x2":1355,"y2":1360,"stroke":70,"style":1338},"102","146",[46,1362],{"x":1352,"y":83,"width":153,"height":1329,"rx":61,"fill":51,"stroke":104,"style":87},[72,1364,1365],{"x":1355,"y":157,"fill":76,"style":173},"Pydantic validate",[95,1367],{"x1":1352,"y1":1368,"x2":1369,"y2":1368,"stroke":70,"style":1338},"177","445",[46,1371],{"x":1341,"y":83,"width":1342,"height":1329,"rx":61,"fill":51,"stroke":1372,"style":87},"var(--c-yellow)",[72,1374,1375],{"x":1345,"y":157,"fill":76,"style":173},"Repair once, then stop",[95,1377],{"x1":1341,"y1":1368,"x2":135,"y2":1368,"stroke":70,"style":1338},[46,1379],{"x":81,"y":83,"width":1328,"height":1329,"rx":61,"fill":51,"stroke":116,"style":87},[72,1381,1382],{"x":91,"y":157,"fill":76,"style":173},"Dead-letter",[95,1384],{"x1":1355,"y1":1385,"x2":1355,"y2":1386,"stroke":70,"style":1338},"204","228",[46,1388],{"x":1352,"y":1389,"width":153,"height":1390,"rx":61,"fill":149,"stroke":150,"style":87},"232","38",[72,1392,1394],{"x":1355,"y":1393,"fill":76,"style":173},"256","Persist typed row",[72,1396,1399],{"x":1397,"y":1398,"fill":70,"style":137},"455","168","invalid",[72,1401,1404],{"x":1402,"y":1403,"fill":70,"style":137},"600","222","valid",[176,1406],{},[179,1408,1410],{"id":1409},"step-3-batch-with-a-semaphore-not-with-a-thread-pool","Step 3: Batch with a semaphore, not with a thread pool",[14,1412,1413],{},"Model calls are almost pure network wait. One job spends fifty seconds doing nothing but holding a socket open. That makes async the obvious concurrency model — you can hold sixteen of those sockets in a single process with essentially no CPU cost, where sixteen threads would cost you sixteen stacks and a GIL argument.",[14,1415,1416,1417,1420,1421,1424],{},"The rule is: fan out with ",[187,1418,1419],{},"asyncio.gather",", but gate the fan-out with a semaphore. Unbounded ",[187,1422,1423],{},"gather"," over a thousand queue items sends a thousand simultaneous requests, trips the provider's rate limit within a second, and turns your whole batch into a retry storm. The semaphore is what converts \"as fast as possible\" into \"as fast as allowed\".",[204,1426,1428],{"className":349,"code":1427,"language":351,"meta":209,"style":209},"# llm\u002Fbatch.py\nimport asyncio\nimport os\nfrom dataclasses import dataclass\n\nfrom pydantic import ValidationError\n\nfrom .extract import ExtractionRefused, Lead, extract_lead\n\nCONCURRENCY = int(os.getenv(\"LLM_CONCURRENCY\", \"8\"))\n\n\n@dataclass(slots=True)\nclass JobResult:\n    job_id: str\n    lead: Lead | None\n    error: str | None\n    meta: dict\n\n\nasync def _run_one(sem: asyncio.Semaphore, job_id: str, raw: str) -> JobResult:\n    async with sem:\n        lead, meta = await extract_lead(raw)\n        return JobResult(job_id=job_id, lead=lead, error=None, meta=meta)\n\n\nasync def run_batch(jobs: dict[str, str]) -> list[JobResult]:\n    sem = asyncio.Semaphore(CONCURRENCY)\n    tasks = [_run_one(sem, jid, raw) for jid, raw in jobs.items()]\n    settled = await asyncio.gather(*tasks, return_exceptions=True)\n\n    out: list[JobResult] = []\n    for job_id, result in zip(jobs, settled, strict=True):\n        match result:\n            case JobResult():\n                out.append(result)\n            case ExtractionRefused() as exc:\n                out.append(JobResult(job_id, None, f\"refused: {exc}\", {}))\n            case ValidationError() as exc:\n                out.append(JobResult(job_id, None, f\"invalid: {exc.error_count()}\", {}))\n            case BaseException() as exc:\n                out.append(JobResult(job_id, None, f\"{type(exc).__name__}: {exc}\", {}))\n    return out\n",[187,1429,1430,1435,1442,1448,1460,1464,1475,1479,1491,1495,1516,1520,1524,1541,1550,1557,1567,1578,1586,1590,1594,1616,1627,1639,1680,1684,1688,1709,1723,1743,1770,1774,1784,1809,1817,1825,1830,1843,1870,1881,1905,1919,1952],{"__ignoreMap":209},[213,1431,1432],{"class":95,"line":215},[213,1433,1434],{"class":358},"# llm\u002Fbatch.py\n",[213,1436,1437,1439],{"class":95,"line":264},[213,1438,365],{"class":364},[213,1440,1441],{"class":368}," asyncio\n",[213,1443,1444,1446],{"class":95,"line":270},[213,1445,365],{"class":364},[213,1447,369],{"class":368},[213,1449,1450,1452,1455,1457],{"class":95,"line":276},[213,1451,374],{"class":364},[213,1453,1454],{"class":368}," dataclasses ",[213,1456,365],{"class":364},[213,1458,1459],{"class":368}," dataclass\n",[213,1461,1462],{"class":95,"line":282},[213,1463,388],{"emptyLinePlaceholder":387},[213,1465,1466,1468,1470,1472],{"class":95,"line":288},[213,1467,374],{"class":364},[213,1469,647],{"class":368},[213,1471,365],{"class":364},[213,1473,1474],{"class":368}," ValidationError\n",[213,1476,1477],{"class":95,"line":294},[213,1478,388],{"emptyLinePlaceholder":387},[213,1480,1481,1483,1486,1488],{"class":95,"line":300},[213,1482,374],{"class":364},[213,1484,1485],{"class":368}," .extract ",[213,1487,365],{"class":364},[213,1489,1490],{"class":368}," ExtractionRefused, Lead, extract_lead\n",[213,1492,1493],{"class":95,"line":306},[213,1494,388],{"emptyLinePlaceholder":387},[213,1496,1497,1500,1502,1504,1506,1509,1511,1514],{"class":95,"line":312},[213,1498,1499],{"class":409},"CONCURRENCY",[213,1501,413],{"class":364},[213,1503,457],{"class":409},[213,1505,460],{"class":368},[213,1507,1508],{"class":222},"\"LLM_CONCURRENCY\"",[213,1510,422],{"class":368},[213,1512,1513],{"class":222},"\"8\"",[213,1515,471],{"class":368},[213,1517,1518],{"class":95,"line":318},[213,1519,388],{"emptyLinePlaceholder":387},[213,1521,1522],{"class":95,"line":500},[213,1523,388],{"emptyLinePlaceholder":387},[213,1525,1526,1529,1531,1534,1536,1539],{"class":95,"line":505},[213,1527,1528],{"class":218},"@dataclass",[213,1530,511],{"class":368},[213,1532,1533],{"class":514},"slots",[213,1535,518],{"class":364},[213,1537,1538],{"class":409},"True",[213,1540,428],{"class":368},[213,1542,1543,1545,1548],{"class":95,"line":526},[213,1544,700],{"class":364},[213,1546,1547],{"class":218}," JobResult",[213,1549,1114],{"class":368},[213,1551,1552,1555],{"class":95,"line":538},[213,1553,1554],{"class":368},"    job_id: ",[213,1556,719],{"class":409},[213,1558,1559,1562,1565],{"class":95,"line":544},[213,1560,1561],{"class":368},"    lead: Lead ",[213,1563,1564],{"class":364},"|",[213,1566,745],{"class":409},[213,1568,1569,1572,1574,1576],{"class":95,"line":553},[213,1570,1571],{"class":368},"    error: ",[213,1573,899],{"class":409},[213,1575,737],{"class":364},[213,1577,745],{"class":409},[213,1579,1580,1583],{"class":95,"line":569},[213,1581,1582],{"class":368},"    meta: ",[213,1584,1585],{"class":409},"dict\n",[213,1587,1588],{"class":95,"line":585},[213,1589,388],{"emptyLinePlaceholder":387},[213,1591,1592],{"class":95,"line":813},[213,1593,388],{"emptyLinePlaceholder":387},[213,1595,1596,1598,1600,1603,1606,1608,1611,1613],{"class":95,"line":818},[213,1597,887],{"class":364},[213,1599,890],{"class":364},[213,1601,1602],{"class":218}," _run_one",[213,1604,1605],{"class":368},"(sem: asyncio.Semaphore, job_id: ",[213,1607,899],{"class":409},[213,1609,1610],{"class":368},", raw: ",[213,1612,899],{"class":409},[213,1614,1615],{"class":368},") -> JobResult:\n",[213,1617,1618,1621,1624],{"class":95,"line":823},[213,1619,1620],{"class":364},"    async",[213,1622,1623],{"class":364}," with",[213,1625,1626],{"class":368}," sem:\n",[213,1628,1629,1632,1634,1636],{"class":95,"line":838},[213,1630,1631],{"class":368},"        lead, meta ",[213,1633,518],{"class":364},[213,1635,930],{"class":364},[213,1637,1638],{"class":368}," extract_lead(raw)\n",[213,1640,1641,1644,1647,1650,1652,1655,1658,1660,1663,1666,1668,1670,1672,1675,1677],{"class":95,"line":844},[213,1642,1643],{"class":364},"        return",[213,1645,1646],{"class":368}," JobResult(",[213,1648,1649],{"class":514},"job_id",[213,1651,518],{"class":364},[213,1653,1654],{"class":368},"job_id, ",[213,1656,1657],{"class":514},"lead",[213,1659,518],{"class":364},[213,1661,1662],{"class":368},"lead, ",[213,1664,1665],{"class":514},"error",[213,1667,518],{"class":364},[213,1669,1141],{"class":409},[213,1671,422],{"class":368},[213,1673,1674],{"class":514},"meta",[213,1676,518],{"class":364},[213,1678,1679],{"class":368},"meta)\n",[213,1681,1682],{"class":95,"line":849},[213,1683,388],{"emptyLinePlaceholder":387},[213,1685,1686],{"class":95,"line":854},[213,1687,388],{"emptyLinePlaceholder":387},[213,1689,1690,1692,1694,1697,1700,1702,1704,1706],{"class":95,"line":868},[213,1691,887],{"class":364},[213,1693,890],{"class":364},[213,1695,1696],{"class":218}," run_batch",[213,1698,1699],{"class":368},"(jobs: dict[",[213,1701,899],{"class":409},[213,1703,422],{"class":368},[213,1705,899],{"class":409},[213,1707,1708],{"class":368},"]) -> list[JobResult]:\n",[213,1710,1711,1714,1716,1719,1721],{"class":95,"line":874},[213,1712,1713],{"class":368},"    sem ",[213,1715,518],{"class":364},[213,1717,1718],{"class":368}," asyncio.Semaphore(",[213,1720,1499],{"class":409},[213,1722,428],{"class":368},[213,1724,1725,1728,1730,1733,1735,1738,1740],{"class":95,"line":879},[213,1726,1727],{"class":368},"    tasks ",[213,1729,518],{"class":364},[213,1731,1732],{"class":368}," [_run_one(sem, jid, raw) ",[213,1734,1198],{"class":364},[213,1736,1737],{"class":368}," jid, raw ",[213,1739,1204],{"class":364},[213,1741,1742],{"class":368}," jobs.items()]\n",[213,1744,1745,1748,1750,1752,1755,1758,1761,1764,1766,1768],{"class":95,"line":884},[213,1746,1747],{"class":368},"    settled ",[213,1749,518],{"class":364},[213,1751,930],{"class":364},[213,1753,1754],{"class":368}," asyncio.gather(",[213,1756,1757],{"class":364},"*",[213,1759,1760],{"class":368},"tasks, ",[213,1762,1763],{"class":514},"return_exceptions",[213,1765,518],{"class":364},[213,1767,1538],{"class":409},[213,1769,428],{"class":368},[213,1771,1772],{"class":95,"line":911},[213,1773,388],{"emptyLinePlaceholder":387},[213,1775,1776,1779,1781],{"class":95,"line":922},[213,1777,1778],{"class":368},"    out: list[JobResult] ",[213,1780,518],{"class":364},[213,1782,1783],{"class":368}," []\n",[213,1785,1786,1789,1792,1794,1797,1800,1803,1805,1807],{"class":95,"line":936},[213,1787,1788],{"class":364},"    for",[213,1790,1791],{"class":368}," job_id, result ",[213,1793,1204],{"class":364},[213,1795,1796],{"class":409}," zip",[213,1798,1799],{"class":368},"(jobs, settled, ",[213,1801,1802],{"class":514},"strict",[213,1804,518],{"class":364},[213,1806,1538],{"class":409},[213,1808,711],{"class":368},[213,1810,1811,1814],{"class":95,"line":949},[213,1812,1813],{"class":364},"        match",[213,1815,1816],{"class":368}," result:\n",[213,1818,1819,1822],{"class":95,"line":961},[213,1820,1821],{"class":364},"            case",[213,1823,1824],{"class":368}," JobResult():\n",[213,1826,1827],{"class":95,"line":972},[213,1828,1829],{"class":368},"                out.append(result)\n",[213,1831,1832,1834,1837,1840],{"class":95,"line":986},[213,1833,1821],{"class":364},[213,1835,1836],{"class":368}," ExtractionRefused() ",[213,1838,1839],{"class":364},"as",[213,1841,1842],{"class":368}," exc:\n",[213,1844,1845,1848,1850,1852,1854,1857,1859,1862,1865,1867],{"class":95,"line":995},[213,1846,1847],{"class":368},"                out.append(JobResult(job_id, ",[213,1849,1141],{"class":409},[213,1851,422],{"class":368},[213,1853,1165],{"class":364},[213,1855,1856],{"class":222},"\"refused: ",[213,1858,1032],{"class":409},[213,1860,1861],{"class":368},"exc",[213,1863,1864],{"class":409},"}",[213,1866,1174],{"class":222},[213,1868,1869],{"class":368},", {}))\n",[213,1871,1872,1874,1877,1879],{"class":95,"line":1018},[213,1873,1821],{"class":364},[213,1875,1876],{"class":368}," ValidationError() ",[213,1878,1839],{"class":364},[213,1880,1842],{"class":368},[213,1882,1883,1885,1887,1889,1891,1894,1896,1899,1901,1903],{"class":95,"line":1024},[213,1884,1847],{"class":368},[213,1886,1141],{"class":409},[213,1888,422],{"class":368},[213,1890,1165],{"class":364},[213,1892,1893],{"class":222},"\"invalid: ",[213,1895,1032],{"class":409},[213,1897,1898],{"class":368},"exc.error_count()",[213,1900,1864],{"class":409},[213,1902,1174],{"class":222},[213,1904,1869],{"class":368},[213,1906,1907,1909,1912,1915,1917],{"class":95,"line":1059},[213,1908,1821],{"class":364},[213,1910,1911],{"class":409}," BaseException",[213,1913,1914],{"class":368},"() ",[213,1916,1839],{"class":364},[213,1918,1842],{"class":368},[213,1920,1921,1923,1925,1927,1929,1931,1934,1937,1940,1942,1944,1946,1948,1950],{"class":95,"line":1086},[213,1922,1847],{"class":368},[213,1924,1141],{"class":409},[213,1926,422],{"class":368},[213,1928,1165],{"class":364},[213,1930,1174],{"class":222},[213,1932,1933],{"class":409},"{type",[213,1935,1936],{"class":368},"(exc).",[213,1938,1939],{"class":409},"__name__}",[213,1941,978],{"class":222},[213,1943,1032],{"class":409},[213,1945,1861],{"class":368},[213,1947,1864],{"class":409},[213,1949,1174],{"class":222},[213,1951,1869],{"class":368},[213,1953,1954,1956],{"class":95,"line":1091},[213,1955,547],{"class":364},[213,1957,1958],{"class":368}," out\n",[14,1960,1961,1964,1965,1967],{},[187,1962,1963],{},"return_exceptions=True"," is what makes this batch survivable: one poisoned document does not cancel the other nine hundred and ninety-nine in flight. The ",[187,1966,189],{}," statement then sorts outcomes into buckets you can act on — refusals go to a human, validation failures go to a repair pass, everything else goes to the retry policy.",[14,1969,1970,1971,1975],{},"Pick the concurrency number from your provider's tokens-per-minute limit, not from your CPU count. If your account allows 400,000 input tokens per minute and each job sends 6,000, you can sustain roughly 66 jobs per minute; at 50 seconds per job that is about 55 concurrent calls before you saturate. Start at a quarter of your computed ceiling and raise it while watching your 429 rate. The mechanics of reading those headers are covered in ",[18,1972,1974],{"href":1973},"\u002Fgetting-started-with-python-apis-for-builders\u002Fmaking-http-requests-with-requests-library\u002Fdebugging-429-too-many-requests-errors\u002F","debugging 429 Too Many Requests errors",".",[176,1977],{},[179,1979,1981],{"id":1980},"step-4-retry-the-transients-kill-the-permanents","Step 4: Retry the transients, kill the permanents",[14,1983,1984],{},"Retry logic around a model call is different from retry logic around a normal REST call in one crucial way: a failed attempt may still have cost you money. A request that times out after fifty seconds of generation is billed for the tokens produced before you hung up. So the question is not \"is this error retryable\" but \"is retrying this cheaper than failing\".",[14,1986,1987],{},"Three classes of outcome exist. Rate limits, connection errors, and 5xx responses are transient and worth retrying with exponential backoff and jitter. A 400 is a bug in your request and will fail identically forever. A refusal is a policy decision and will also fail identically forever, but unlike a 400 it costs you tokens each time you ask. Retrying refusals is the most expensive no-op in the business.",[204,1989,1991],{"className":349,"code":1990,"language":351,"meta":209,"style":209},"# llm\u002Fretry.py\nimport os\nimport random\n\nimport anthropic\nfrom tenacity import (\n    AsyncRetrying,\n    RetryError,\n    retry_if_exception_type,\n    stop_after_attempt,\n    wait_exponential_jitter,\n)\n\nMAX_ATTEMPTS = int(os.getenv(\"LLM_MAX_ATTEMPTS\", \"4\"))\n\nTRANSIENT = (\n    anthropic.RateLimitError,        # 429\n    anthropic.InternalServerError,   # 500 and 529 overloaded\n    anthropic.APIConnectionError,    # DNS, TLS, read timeout\n)\n\n\ndef _retry_after_seconds(exc: BaseException) -> float | None:\n    response = getattr(exc, \"response\", None)\n    header = getattr(response, \"headers\", {}).get(\"retry-after\") if response else None\n    try:\n        return float(header) if header else None\n    except ValueError:\n        return None\n\n\nasync def with_retry(coro_factory):\n    \"\"\"Run coro_factory() with backoff. Non-transient errors propagate at once.\"\"\"\n    async for attempt in AsyncRetrying(\n        retry=retry_if_exception_type(TRANSIENT),\n        stop=stop_after_attempt(MAX_ATTEMPTS),\n        wait=wait_exponential_jitter(initial=1, max=30, jitter=2),\n        reraise=True,\n    ):\n        with attempt:\n            return await coro_factory()\n",[187,1992,1993,1998,2004,2011,2015,2022,2034,2039,2044,2049,2054,2059,2063,2067,2088,2092,2101,2109,2117,2125,2129,2133,2137,2162,2184,2218,2225,2243,2253,2259,2263,2267,2279,2284,2299,2314,2328,2367,2378,2383,2391],{"__ignoreMap":209},[213,1994,1995],{"class":95,"line":215},[213,1996,1997],{"class":358},"# llm\u002Fretry.py\n",[213,1999,2000,2002],{"class":95,"line":264},[213,2001,365],{"class":364},[213,2003,369],{"class":368},[213,2005,2006,2008],{"class":95,"line":270},[213,2007,365],{"class":364},[213,2009,2010],{"class":368}," random\n",[213,2012,2013],{"class":95,"line":276},[213,2014,388],{"emptyLinePlaceholder":387},[213,2016,2017,2019],{"class":95,"line":282},[213,2018,365],{"class":364},[213,2020,2021],{"class":368}," anthropic\n",[213,2023,2024,2026,2029,2031],{"class":95,"line":288},[213,2025,374],{"class":364},[213,2027,2028],{"class":368}," tenacity ",[213,2030,365],{"class":364},[213,2032,2033],{"class":368}," (\n",[213,2035,2036],{"class":95,"line":294},[213,2037,2038],{"class":368},"    AsyncRetrying,\n",[213,2040,2041],{"class":95,"line":300},[213,2042,2043],{"class":368},"    RetryError,\n",[213,2045,2046],{"class":95,"line":306},[213,2047,2048],{"class":368},"    retry_if_exception_type,\n",[213,2050,2051],{"class":95,"line":312},[213,2052,2053],{"class":368},"    stop_after_attempt,\n",[213,2055,2056],{"class":95,"line":318},[213,2057,2058],{"class":368},"    wait_exponential_jitter,\n",[213,2060,2061],{"class":95,"line":500},[213,2062,428],{"class":368},[213,2064,2065],{"class":95,"line":505},[213,2066,388],{"emptyLinePlaceholder":387},[213,2068,2069,2072,2074,2076,2078,2081,2083,2086],{"class":95,"line":526},[213,2070,2071],{"class":409},"MAX_ATTEMPTS",[213,2073,413],{"class":364},[213,2075,457],{"class":409},[213,2077,460],{"class":368},[213,2079,2080],{"class":222},"\"LLM_MAX_ATTEMPTS\"",[213,2082,422],{"class":368},[213,2084,2085],{"class":222},"\"4\"",[213,2087,471],{"class":368},[213,2089,2090],{"class":95,"line":538},[213,2091,388],{"emptyLinePlaceholder":387},[213,2093,2094,2097,2099],{"class":95,"line":544},[213,2095,2096],{"class":409},"TRANSIENT",[213,2098,413],{"class":364},[213,2100,2033],{"class":368},[213,2102,2103,2106],{"class":95,"line":553},[213,2104,2105],{"class":368},"    anthropic.RateLimitError,        ",[213,2107,2108],{"class":358},"# 429\n",[213,2110,2111,2114],{"class":95,"line":569},[213,2112,2113],{"class":368},"    anthropic.InternalServerError,   ",[213,2115,2116],{"class":358},"# 500 and 529 overloaded\n",[213,2118,2119,2122],{"class":95,"line":585},[213,2120,2121],{"class":368},"    anthropic.APIConnectionError,    ",[213,2123,2124],{"class":358},"# DNS, TLS, read timeout\n",[213,2126,2127],{"class":95,"line":813},[213,2128,428],{"class":368},[213,2130,2131],{"class":95,"line":818},[213,2132,388],{"emptyLinePlaceholder":387},[213,2134,2135],{"class":95,"line":823},[213,2136,388],{"emptyLinePlaceholder":387},[213,2138,2139,2141,2144,2147,2150,2153,2156,2158,2160],{"class":95,"line":838},[213,2140,529],{"class":364},[213,2142,2143],{"class":218}," _retry_after_seconds",[213,2145,2146],{"class":368},"(exc: ",[213,2148,2149],{"class":409},"BaseException",[213,2151,2152],{"class":368},") -> ",[213,2154,2155],{"class":409},"float",[213,2157,737],{"class":364},[213,2159,740],{"class":409},[213,2161,1114],{"class":368},[213,2163,2164,2167,2169,2172,2175,2178,2180,2182],{"class":95,"line":844},[213,2165,2166],{"class":368},"    response ",[213,2168,518],{"class":364},[213,2170,2171],{"class":409}," getattr",[213,2173,2174],{"class":368},"(exc, ",[213,2176,2177],{"class":222},"\"response\"",[213,2179,422],{"class":368},[213,2181,1141],{"class":409},[213,2183,428],{"class":368},[213,2185,2186,2189,2191,2193,2196,2199,2202,2205,2208,2210,2213,2216],{"class":95,"line":849},[213,2187,2188],{"class":368},"    header ",[213,2190,518],{"class":364},[213,2192,2171],{"class":409},[213,2194,2195],{"class":368},"(response, ",[213,2197,2198],{"class":222},"\"headers\"",[213,2200,2201],{"class":368},", {}).get(",[213,2203,2204],{"class":222},"\"retry-after\"",[213,2206,2207],{"class":368},") ",[213,2209,1210],{"class":364},[213,2211,2212],{"class":368}," response ",[213,2214,2215],{"class":364},"else",[213,2217,745],{"class":409},[213,2219,2220,2223],{"class":95,"line":854},[213,2221,2222],{"class":364},"    try",[213,2224,1114],{"class":368},[213,2226,2227,2229,2231,2234,2236,2239,2241],{"class":95,"line":868},[213,2228,1643],{"class":364},[213,2230,481],{"class":409},[213,2232,2233],{"class":368},"(header) ",[213,2235,1210],{"class":364},[213,2237,2238],{"class":368}," header ",[213,2240,2215],{"class":364},[213,2242,745],{"class":409},[213,2244,2245,2248,2251],{"class":95,"line":874},[213,2246,2247],{"class":364},"    except",[213,2249,2250],{"class":409}," ValueError",[213,2252,1114],{"class":368},[213,2254,2255,2257],{"class":95,"line":879},[213,2256,1643],{"class":364},[213,2258,745],{"class":409},[213,2260,2261],{"class":95,"line":884},[213,2262,388],{"emptyLinePlaceholder":387},[213,2264,2265],{"class":95,"line":911},[213,2266,388],{"emptyLinePlaceholder":387},[213,2268,2269,2271,2273,2276],{"class":95,"line":922},[213,2270,887],{"class":364},[213,2272,890],{"class":364},[213,2274,2275],{"class":218}," with_retry",[213,2277,2278],{"class":368},"(coro_factory):\n",[213,2280,2281],{"class":95,"line":936},[213,2282,2283],{"class":222},"    \"\"\"Run coro_factory() with backoff. Non-transient errors propagate at once.\"\"\"\n",[213,2285,2286,2288,2291,2294,2296],{"class":95,"line":949},[213,2287,1620],{"class":364},[213,2289,2290],{"class":364}," for",[213,2292,2293],{"class":368}," attempt ",[213,2295,1204],{"class":364},[213,2297,2298],{"class":368}," AsyncRetrying(\n",[213,2300,2301,2304,2306,2309,2311],{"class":95,"line":961},[213,2302,2303],{"class":514},"        retry",[213,2305,518],{"class":364},[213,2307,2308],{"class":368},"retry_if_exception_type(",[213,2310,2096],{"class":409},[213,2312,2313],{"class":368},"),\n",[213,2315,2316,2319,2321,2324,2326],{"class":95,"line":972},[213,2317,2318],{"class":514},"        stop",[213,2320,518],{"class":364},[213,2322,2323],{"class":368},"stop_after_attempt(",[213,2325,2071],{"class":409},[213,2327,2313],{"class":368},[213,2329,2330,2333,2335,2338,2341,2343,2345,2347,2350,2352,2355,2357,2360,2362,2365],{"class":95,"line":986},[213,2331,2332],{"class":514},"        wait",[213,2334,518],{"class":364},[213,2336,2337],{"class":368},"wait_exponential_jitter(",[213,2339,2340],{"class":514},"initial",[213,2342,518],{"class":364},[213,2344,521],{"class":409},[213,2346,422],{"class":368},[213,2348,2349],{"class":514},"max",[213,2351,518],{"class":364},[213,2353,2354],{"class":409},"30",[213,2356,422],{"class":368},[213,2358,2359],{"class":514},"jitter",[213,2361,518],{"class":364},[213,2363,2364],{"class":409},"2",[213,2366,2313],{"class":368},[213,2368,2369,2372,2374,2376],{"class":95,"line":995},[213,2370,2371],{"class":514},"        reraise",[213,2373,518],{"class":364},[213,2375,1538],{"class":409},[213,2377,946],{"class":368},[213,2379,2380],{"class":95,"line":1018},[213,2381,2382],{"class":368},"    ):\n",[213,2384,2385,2388],{"class":95,"line":1024},[213,2386,2387],{"class":364},"        with",[213,2389,2390],{"class":368}," attempt:\n",[213,2392,2393,2396,2398],{"class":95,"line":1059},[213,2394,2395],{"class":364},"            return",[213,2397,930],{"class":364},[213,2399,2400],{"class":368}," coro_factory()\n",[14,2402,2403,2404,2407,2408,1975],{},"Wrap the extraction call with ",[187,2405,2406],{},"with_retry(lambda: extract_lead(raw))"," and the policy applies to every job without touching the extraction code. The jitter matters more than the backoff: without it, sixteen workers that all hit a rate limit at the same instant will all wake up at the same instant and hit it again. The general pattern, including the decorator form, is covered in ",[18,2409,2411],{"href":2410},"\u002Fgetting-started-with-python-apis-for-builders\u002Fmaking-http-requests-with-requests-library\u002Fretrying-failed-http-requests-with-tenacity\u002F","retrying failed HTTP requests with tenacity",[27,2413,36,2418,36,2421,36,2424,36,2427,36,2434,36,2438,36,2442,36,2447,36,2450,36,2454,36,2458,36,2461,36,2464,36,2468,36,2473,36,2477,36,2479,36,2484,36,2486,36,2489,36,2492,36,2495,36,2499,36,2503,36,2507,36,2509,36,2511,36,2514,36,2516,36,2518,36,2521],{"viewBox":2414,"role":30,"ariaLabelledBy":2415,"xmlns":34,"style":35},"0 0 720 345",[2416,2417],"aiwf-retry-t","aiwf-retry-d",[38,2419,2420],{"id":2416},"Decision tree classifying a model call outcome into retry, commit, or dead-letter",[42,2422,2423],{"id":2417},"A model call outcome is classified three ways: rate limits, server errors and timeouts go to backoff and are retried; a valid schema-conforming response is committed; 4xx errors and refusals go straight to a dead-letter queue with an alert.",[46,2425],{"x":48,"y":48,"width":1311,"height":2426,"fill":51},"345",[53,2428,55,2429,36],{},[57,2430,65,2432,55],{"id":2431,"viewBox":60,"refX":61,"refY":62,"markerWidth":63,"markerHeight":63,"orient":64},"aiwf-retry-arrow",[67,2433],{"d":69,"fill":70},[46,2435],{"x":2436,"y":2437,"width":99,"height":1327,"rx":61,"fill":51,"stroke":86,"style":87},"260","24",[72,2439,2441],{"x":1322,"y":2440,"fill":76,"style":173},"53","Issue model call",[95,2443],{"x1":1322,"y1":2444,"x2":1322,"y2":2445,"stroke":70,"style":2446},"72","104","stroke-width:2;marker-end:url(#aiwf-retry-arrow);",[46,2448],{"x":2436,"y":2449,"width":99,"height":1327,"rx":61,"fill":51,"stroke":70,"style":87},"108",[72,2451,2453],{"x":1322,"y":2452,"fill":76,"style":173},"137","Classify outcome",[67,2455],{"d":2456,"fill":2457,"stroke":70,"style":2446},"M360 156 L360 180 L120 180 L120 200","none",[67,2459],{"d":2460,"fill":2457,"stroke":70,"style":2446},"M360 156 L360 200",[67,2462],{"d":2463,"fill":2457,"stroke":70,"style":2446},"M360 180 L600 180 L600 200",[72,2465,2467],{"x":99,"y":2466,"fill":70,"style":137},"174","transient",[72,2469,2472],{"x":2470,"y":2471,"fill":70,"style":137},"404","196","ok",[72,2474,2476],{"x":2475,"y":2466,"fill":70,"style":137},"524","permanent",[46,2478],{"x":81,"y":1385,"width":99,"height":1327,"rx":61,"fill":51,"stroke":1372,"style":87},[72,2480,2483],{"x":2481,"y":2482,"fill":76,"style":173},"120","233","429 \u002F 5xx \u002F timeout",[46,2485],{"x":2436,"y":1385,"width":99,"height":1327,"rx":61,"fill":51,"stroke":104,"style":87},[72,2487,2488],{"x":1322,"y":2482,"fill":76,"style":173},"2xx + schema valid",[46,2490],{"x":2491,"y":1385,"width":99,"height":1327,"rx":61,"fill":51,"stroke":116,"style":87},"500",[72,2493,2494],{"x":1402,"y":2482,"fill":76,"style":173},"4xx \u002F refusal",[95,2496],{"x1":2481,"y1":2497,"x2":2481,"y2":2498,"stroke":70,"style":2446},"252","272",[46,2500],{"x":81,"y":2501,"width":99,"height":2502,"rx":61,"fill":149,"stroke":150,"style":87},"276","44",[72,2504,2506],{"x":2481,"y":2505,"fill":76,"style":173},"303","Backoff, retry to N",[95,2508],{"x1":1322,"y1":2497,"x2":1322,"y2":2498,"stroke":70,"style":2446},[46,2510],{"x":2436,"y":2501,"width":99,"height":2502,"rx":61,"fill":149,"stroke":150,"style":87},[72,2512,2513],{"x":1322,"y":2505,"fill":76,"style":173},"Commit result",[95,2515],{"x1":1402,"y1":2497,"x2":1402,"y2":2498,"stroke":70,"style":2446},[46,2517],{"x":2491,"y":2501,"width":99,"height":2502,"rx":61,"fill":149,"stroke":150,"style":87},[72,2519,2520],{"x":1402,"y":2505,"fill":76,"style":173},"Dead-letter, alert",[67,2522],{"d":2523,"fill":2457,"stroke":86,"style":2524},"M20 298 L8 298 L8 48 L256 48","stroke-width:2;stroke-dasharray:5 4;marker-end:url(#aiwf-retry-arrow);",[176,2526],{},[179,2528,2530],{"id":2529},"step-5-version-prompts-like-code-because-they-are-code","Step 5: Version prompts like code, because they are code",[14,2532,2533,2534,2537],{},"A prompt is the most volatile piece of logic in the pipeline and the only one most teams edit without review. Change one sentence and your extraction accuracy moves three points in a direction nobody measures until a customer complains. Worse, an inline prompt string interpolated with ",[187,2535,2536],{},"datetime.now()"," or a job ID silently destroys prompt caching, because caching is a prefix match and any byte change invalidates everything after it.",[14,2539,2540,2541,2543],{},"Store prompts in a TOML file, keyed by version, loaded through ",[187,2542,193],{},". Hash the text and attach the digest to every log line and every stored row. Then rolling back a bad prompt is one environment variable, not a deploy.",[204,2545,2549],{"className":2546,"code":2547,"language":2548,"meta":209,"style":209},"language-toml shiki shiki-themes github-light github-dark","# prompts\u002Flead_extract.toml\n[v2]\nreleased = \"2026-06-02\"\nsystem = \"\"\"\nYou extract sales-lead fields from inbound email text.\nReturn only fields you can support with a direct quote from the input.\n\"\"\"\n\n[v3]\nreleased = \"2026-07-14\"\nsystem = \"\"\"\nYou extract sales-lead fields from inbound email text.\nReturn only fields you can support with a direct quote from the input.\nIf a budget figure appears as a range, record the lower bound.\nNever infer an intent that is not stated; use \"other\" when unsure.\n\"\"\"\n","toml",[187,2550,2551,2556,2565,2573,2581,2586,2591,2595,2599,2608,2615,2621,2625,2629,2634,2639],{"__ignoreMap":209},[213,2552,2553],{"class":95,"line":215},[213,2554,2555],{"class":358},"# prompts\u002Flead_extract.toml\n",[213,2557,2558,2560,2563],{"class":95,"line":264},[213,2559,796],{"class":368},[213,2561,2562],{"class":218},"v2",[213,2564,771],{"class":368},[213,2566,2567,2570],{"class":95,"line":270},[213,2568,2569],{"class":368},"released = ",[213,2571,2572],{"class":222},"\"2026-06-02\"\n",[213,2574,2575,2578],{"class":95,"line":276},[213,2576,2577],{"class":368},"system = ",[213,2579,2580],{"class":222},"\"\"\"\n",[213,2582,2583],{"class":95,"line":282},[213,2584,2585],{"class":222},"You extract sales-lead fields from inbound email text.\n",[213,2587,2588],{"class":95,"line":288},[213,2589,2590],{"class":222},"Return only fields you can support with a direct quote from the input.\n",[213,2592,2593],{"class":95,"line":294},[213,2594,2580],{"class":222},[213,2596,2597],{"class":95,"line":300},[213,2598,388],{"emptyLinePlaceholder":387},[213,2600,2601,2603,2606],{"class":95,"line":306},[213,2602,796],{"class":368},[213,2604,2605],{"class":218},"v3",[213,2607,771],{"class":368},[213,2609,2610,2612],{"class":95,"line":312},[213,2611,2569],{"class":368},[213,2613,2614],{"class":222},"\"2026-07-14\"\n",[213,2616,2617,2619],{"class":95,"line":318},[213,2618,2577],{"class":368},[213,2620,2580],{"class":222},[213,2622,2623],{"class":95,"line":500},[213,2624,2585],{"class":222},[213,2626,2627],{"class":95,"line":505},[213,2628,2590],{"class":222},[213,2630,2631],{"class":95,"line":526},[213,2632,2633],{"class":222},"If a budget figure appears as a range, record the lower bound.\n",[213,2635,2636],{"class":95,"line":538},[213,2637,2638],{"class":222},"Never infer an intent that is not stated; use \"other\" when unsure.\n",[213,2640,2641],{"class":95,"line":544},[213,2642,2580],{"class":222},[204,2644,2646],{"className":349,"code":2645,"language":351,"meta":209,"style":209},"# llm\u002Fprompts.py\nimport hashlib\nimport os\nimport tomllib\nfrom functools import lru_cache\nfrom pathlib import Path\n\nPROMPT_FILE = Path(os.getenv(\"PROMPT_FILE\", \"prompts\u002Flead_extract.toml\"))\nACTIVE_VERSION = os.getenv(\"PROMPT_VERSION\", \"v3\")\n\n\n@lru_cache(maxsize=8)\ndef load_prompt(version: str | None = None) -> tuple[str, str]:\n    \"\"\"Return (system_text, short_sha256) for a prompt version.\"\"\"\n    version = version or ACTIVE_VERSION\n    data = tomllib.loads(PROMPT_FILE.read_text(encoding=\"utf-8\"))\n    if version not in data:\n        raise KeyError(f\"prompt version {version!r} not in {PROMPT_FILE}\")\n    text = data[version][\"system\"].strip()\n    digest = hashlib.sha256(text.encode(\"utf-8\")).hexdigest()[:12]\n    return text, f\"{version}:{digest}\"\n",[187,2647,2648,2653,2660,2666,2673,2683,2695,2699,2719,2738,2742,2746,2761,2792,2797,2813,2838,2854,2889,2904,2924],{"__ignoreMap":209},[213,2649,2650],{"class":95,"line":215},[213,2651,2652],{"class":358},"# llm\u002Fprompts.py\n",[213,2654,2655,2657],{"class":95,"line":264},[213,2656,365],{"class":364},[213,2658,2659],{"class":368}," hashlib\n",[213,2661,2662,2664],{"class":95,"line":270},[213,2663,365],{"class":364},[213,2665,369],{"class":368},[213,2667,2668,2670],{"class":95,"line":276},[213,2669,365],{"class":364},[213,2671,2672],{"class":368}," tomllib\n",[213,2674,2675,2677,2679,2681],{"class":95,"line":282},[213,2676,374],{"class":364},[213,2678,377],{"class":368},[213,2680,365],{"class":364},[213,2682,382],{"class":368},[213,2684,2685,2687,2690,2692],{"class":95,"line":288},[213,2686,374],{"class":364},[213,2688,2689],{"class":368}," pathlib ",[213,2691,365],{"class":364},[213,2693,2694],{"class":368}," Path\n",[213,2696,2697],{"class":95,"line":294},[213,2698,388],{"emptyLinePlaceholder":387},[213,2700,2701,2704,2706,2709,2712,2714,2717],{"class":95,"line":300},[213,2702,2703],{"class":409},"PROMPT_FILE",[213,2705,413],{"class":364},[213,2707,2708],{"class":368}," Path(os.getenv(",[213,2710,2711],{"class":222},"\"PROMPT_FILE\"",[213,2713,422],{"class":368},[213,2715,2716],{"class":222},"\"prompts\u002Flead_extract.toml\"",[213,2718,471],{"class":368},[213,2720,2721,2724,2726,2728,2731,2733,2736],{"class":95,"line":306},[213,2722,2723],{"class":409},"ACTIVE_VERSION",[213,2725,413],{"class":364},[213,2727,416],{"class":368},[213,2729,2730],{"class":222},"\"PROMPT_VERSION\"",[213,2732,422],{"class":368},[213,2734,2735],{"class":222},"\"v3\"",[213,2737,428],{"class":368},[213,2739,2740],{"class":95,"line":312},[213,2741,388],{"emptyLinePlaceholder":387},[213,2743,2744],{"class":95,"line":318},[213,2745,388],{"emptyLinePlaceholder":387},[213,2747,2748,2750,2752,2754,2756,2759],{"class":95,"line":500},[213,2749,508],{"class":218},[213,2751,511],{"class":368},[213,2753,515],{"class":514},[213,2755,518],{"class":364},[213,2757,2758],{"class":409},"8",[213,2760,428],{"class":368},[213,2762,2763,2765,2768,2771,2773,2775,2777,2779,2781,2784,2786,2788,2790],{"class":95,"line":505},[213,2764,529],{"class":364},[213,2766,2767],{"class":218}," load_prompt",[213,2769,2770],{"class":368},"(version: ",[213,2772,899],{"class":409},[213,2774,737],{"class":364},[213,2776,740],{"class":409},[213,2778,413],{"class":364},[213,2780,740],{"class":409},[213,2782,2783],{"class":368},") -> tuple[",[213,2785,899],{"class":409},[213,2787,422],{"class":368},[213,2789,899],{"class":409},[213,2791,908],{"class":368},[213,2793,2794],{"class":95,"line":526},[213,2795,2796],{"class":222},"    \"\"\"Return (system_text, short_sha256) for a prompt version.\"\"\"\n",[213,2798,2799,2802,2804,2807,2810],{"class":95,"line":538},[213,2800,2801],{"class":368},"    version ",[213,2803,518],{"class":364},[213,2805,2806],{"class":368}," version ",[213,2808,2809],{"class":364},"or",[213,2811,2812],{"class":409}," ACTIVE_VERSION\n",[213,2814,2815,2818,2820,2823,2825,2828,2831,2833,2836],{"class":95,"line":544},[213,2816,2817],{"class":368},"    data ",[213,2819,518],{"class":364},[213,2821,2822],{"class":368}," tomllib.loads(",[213,2824,2703],{"class":409},[213,2826,2827],{"class":368},".read_text(",[213,2829,2830],{"class":514},"encoding",[213,2832,518],{"class":364},[213,2834,2835],{"class":222},"\"utf-8\"",[213,2837,471],{"class":368},[213,2839,2840,2843,2845,2848,2851],{"class":95,"line":553},[213,2841,2842],{"class":364},"    if",[213,2844,2806],{"class":368},[213,2846,2847],{"class":364},"not",[213,2849,2850],{"class":364}," in",[213,2852,2853],{"class":368}," data:\n",[213,2855,2856,2859,2862,2864,2866,2869,2871,2874,2877,2879,2882,2885,2887],{"class":95,"line":569},[213,2857,2858],{"class":364},"        raise",[213,2860,2861],{"class":409}," KeyError",[213,2863,511],{"class":368},[213,2865,1165],{"class":364},[213,2867,2868],{"class":222},"\"prompt version ",[213,2870,1032],{"class":409},[213,2872,2873],{"class":368},"version",[213,2875,2876],{"class":364},"!r",[213,2878,1864],{"class":409},[213,2880,2881],{"class":222}," not in ",[213,2883,2884],{"class":409},"{PROMPT_FILE}",[213,2886,1174],{"class":222},[213,2888,428],{"class":368},[213,2890,2891,2893,2895,2898,2901],{"class":95,"line":585},[213,2892,1187],{"class":368},[213,2894,518],{"class":364},[213,2896,2897],{"class":368}," data[version][",[213,2899,2900],{"class":222},"\"system\"",[213,2902,2903],{"class":368},"].strip()\n",[213,2905,2906,2909,2911,2914,2916,2919,2922],{"class":95,"line":813},[213,2907,2908],{"class":368},"    digest ",[213,2910,518],{"class":364},[213,2912,2913],{"class":368}," hashlib.sha256(text.encode(",[213,2915,2835],{"class":222},[213,2917,2918],{"class":368},")).hexdigest()[:",[213,2920,2921],{"class":409},"12",[213,2923,771],{"class":368},[213,2925,2926,2928,2931,2933,2935,2937,2939,2941,2944,2946,2949,2951],{"class":95,"line":818},[213,2927,547],{"class":364},[213,2929,2930],{"class":368}," text, ",[213,2932,1165],{"class":364},[213,2934,1174],{"class":222},[213,2936,1032],{"class":409},[213,2938,2873],{"class":368},[213,2940,1864],{"class":409},[213,2942,2943],{"class":222},":",[213,2945,1032],{"class":409},[213,2947,2948],{"class":368},"digest",[213,2950,1864],{"class":409},[213,2952,2953],{"class":222},"\"\n",[14,2955,2956,2957,2960],{},"Storing ",[187,2958,2959],{},"prompt_digest"," alongside every extracted row is what makes the pipeline auditable. When accuracy drops, you can group your quality metric by digest and see exactly which edit did it. Without that column you are guessing, and guessing about a prompt change three weeks after the fact is an afternoon you never get back.",[27,2962,36,2967,36,2970,36,2973,36,2976,36,2979,36,2983,36,2987,36,2991,36,2995,36,2999,36,3003,36,3005,36,3008,36,3011,36,3014,36,3017,36,3019,36,3022,36,3025,36,3028,36,3031,36,3035],{"viewBox":2963,"role":30,"ariaLabelledBy":2964,"xmlns":34,"style":35},"0 0 720 250",[2965,2966],"aiwf-ver-t","aiwf-ver-d",[38,2968,2969],{"id":2965},"Timeline of prompt versions with measured quality and cost per job",[42,2971,2972],{"id":2966},"Three prompt versions on a timeline, each with a content hash, a valid-output percentage and a cost per job, showing that version three regressed and can be rolled back to version two with an environment variable.",[46,2974],{"x":48,"y":48,"width":1311,"height":2975,"fill":51},"250",[72,2977,2978],{"x":1322,"y":1323,"fill":76,"style":77},"Every prompt edit is a release with a measurable outcome",[95,2980],{"x1":2981,"y1":2481,"x2":2982,"y2":2481,"stroke":150,"style":87},"40","680",[2984,2985],"circle",{"cx":2986,"cy":2481,"r":61,"fill":86},"140",[72,2988,2990],{"x":2986,"y":2989,"fill":70,"style":137},"62","2026-05-11",[72,2992,2994],{"x":2986,"y":2993,"fill":76,"style":92},"84","v1 · 4c1e8a",[72,2996,2998],{"x":2986,"y":2997,"fill":76,"style":173},"152","valid 91.2%",[72,3000,3002],{"x":2986,"y":3001,"fill":70,"style":173},"172","$0.047 \u002F job",[2984,3004],{"cx":1322,"cy":2481,"r":61,"fill":104},[72,3006,3007],{"x":1322,"y":2989,"fill":70,"style":137},"2026-06-02",[72,3009,3010],{"x":1322,"y":2993,"fill":76,"style":92},"v2 · 9b7d02",[72,3012,3013],{"x":1322,"y":2997,"fill":76,"style":173},"valid 98.6%",[72,3015,3016],{"x":1322,"y":3001,"fill":70,"style":173},"$0.025 \u002F job",[2984,3018],{"cx":1355,"cy":2481,"r":61,"fill":116},[72,3020,3021],{"x":1355,"y":2989,"fill":70,"style":137},"2026-07-14",[72,3023,3024],{"x":1355,"y":2993,"fill":76,"style":92},"v3 · e30f57",[72,3026,3027],{"x":1355,"y":2997,"fill":76,"style":173},"valid 96.1%",[72,3029,3030],{"x":1355,"y":3001,"fill":70,"style":173},"$0.026 \u002F job",[67,3032],{"d":3033,"fill":2457,"stroke":70,"style":3034},"M560 196 Q460 226 380 200","stroke-width:2;stroke-dasharray:5 4;",[72,3036,3037],{"x":1322,"y":1386,"fill":70,"style":173},"PROMPT_VERSION=v2 rolls the regression back without a deploy",[176,3039],{},[179,3041,3043],{"id":3042},"step-6-queue-with-a-budget-and-route-the-patient-work-to-batch","Step 6: Queue with a budget, and route the patient work to batch",[14,3045,3046],{},"Now the commercial part. Two levers separate a pipeline that costs $47 per thousand jobs from one that costs $5, and neither of them requires a worse model.",[14,3048,3049,3050,3052,3053,3056],{},"The first lever is prompt caching. Your system prompt, schema, and few-shot examples are identical on every job; only the user document changes. Marking the stable prefix with ",[187,3051,1295],{}," bills those tokens at roughly a tenth of the input rate on cache hits. The prefix must clear a minimum size to cache at all — 4,096 tokens on the larger models — so short system prompts silently do not cache. Check ",[187,3054,3055],{},"usage.cache_read_input_tokens"," on a real response before you believe it works.",[14,3058,3059],{},"The second lever is asynchronous batch submission. Work that does not need an answer this second — nightly enrichment, backfills, a weekly report — can go through the Batch API at half price. Combine both and you are at a quarter of the naive bill.",[204,3061,3063],{"className":349,"code":3062,"language":351,"meta":209,"style":209},"# llm\u002Fbudget.py\nimport os\nfrom datetime import date\n\nimport redis.asyncio as redis\n\nfrom .client import CHEAP_MODEL, MODEL, get_client\n\n# US dollars per million tokens: (input, output)\nPRICES = {\n    \"claude-opus-4-8\": (5.00, 25.00),\n    \"claude-haiku-4-5\": (1.00, 5.00),\n}\nDAILY_BUDGET_USD = float(os.getenv(\"DAILY_BUDGET_USD\", \"25.00\"))\n\n\nclass BudgetExceeded(Exception):\n    \"\"\"Stop the queue rather than discover the overspend on the invoice.\"\"\"\n\n\nasync def estimate_usd(system: list, messages: list, model: str) -> float:\n    counted = await get_client().messages.count_tokens(\n        model=model, system=system, messages=messages\n    )\n    in_rate, out_rate = PRICES[model]\n    projected_out = int(os.getenv(\"LLM_MAX_TOKENS\", \"2048\")) * 0.6\n    return (counted.input_tokens * in_rate + projected_out * out_rate) \u002F 1_000_000\n\n\nasync def reserve(pool: redis.Redis, amount_usd: float) -> None:\n    key = f\"llm:spend:{date.today().isoformat()}\"\n    spent = await pool.incrbyfloat(key, amount_usd)\n    await pool.expire(key, 172_800)\n    if spent > DAILY_BUDGET_USD:\n        await pool.incrbyfloat(key, -amount_usd)\n        raise BudgetExceeded(f\"{spent:.2f} would exceed {DAILY_BUDGET_USD:.2f}\")\n\n\ndef choose_model(input_tokens: int, needs_reasoning: bool) -> str:\n    \"\"\"Route cheap, mechanical work to the small model. Measure before trusting it.\"\"\"\n    return MODEL if needs_reasoning or input_tokens > 20_000 else CHEAP_MODEL\n",[187,3064,3065,3070,3076,3088,3092,3104,3108,3125,3129,3134,3143,3161,3177,3182,3203,3207,3211,3224,3229,3233,3237,3268,3280,3305,3309,3322,3347,3376,3380,3384,3404,3426,3438,3451,3466,3480,3515,3519,3523,3547,3552],{"__ignoreMap":209},[213,3066,3067],{"class":95,"line":215},[213,3068,3069],{"class":358},"# llm\u002Fbudget.py\n",[213,3071,3072,3074],{"class":95,"line":264},[213,3073,365],{"class":364},[213,3075,369],{"class":368},[213,3077,3078,3080,3083,3085],{"class":95,"line":270},[213,3079,374],{"class":364},[213,3081,3082],{"class":368}," datetime ",[213,3084,365],{"class":364},[213,3086,3087],{"class":368}," date\n",[213,3089,3090],{"class":95,"line":276},[213,3091,388],{"emptyLinePlaceholder":387},[213,3093,3094,3096,3099,3101],{"class":95,"line":282},[213,3095,365],{"class":364},[213,3097,3098],{"class":368}," redis.asyncio ",[213,3100,1839],{"class":364},[213,3102,3103],{"class":368}," redis\n",[213,3105,3106],{"class":95,"line":288},[213,3107,388],{"emptyLinePlaceholder":387},[213,3109,3110,3112,3114,3116,3119,3121,3123],{"class":95,"line":294},[213,3111,374],{"class":364},[213,3113,663],{"class":368},[213,3115,365],{"class":364},[213,3117,3118],{"class":409}," CHEAP_MODEL",[213,3120,422],{"class":368},[213,3122,410],{"class":409},[213,3124,675],{"class":368},[213,3126,3127],{"class":95,"line":300},[213,3128,388],{"emptyLinePlaceholder":387},[213,3130,3131],{"class":95,"line":306},[213,3132,3133],{"class":358},"# US dollars per million tokens: (input, output)\n",[213,3135,3136,3139,3141],{"class":95,"line":312},[213,3137,3138],{"class":409},"PRICES",[213,3140,413],{"class":364},[213,3142,1232],{"class":368},[213,3144,3145,3148,3151,3154,3156,3159],{"class":95,"line":318},[213,3146,3147],{"class":222},"    \"claude-opus-4-8\"",[213,3149,3150],{"class":368},": (",[213,3152,3153],{"class":409},"5.00",[213,3155,422],{"class":368},[213,3157,3158],{"class":409},"25.00",[213,3160,2313],{"class":368},[213,3162,3163,3166,3168,3171,3173,3175],{"class":95,"line":500},[213,3164,3165],{"class":222},"    \"claude-haiku-4-5\"",[213,3167,3150],{"class":368},[213,3169,3170],{"class":409},"1.00",[213,3172,422],{"class":368},[213,3174,3153],{"class":409},[213,3176,2313],{"class":368},[213,3178,3179],{"class":95,"line":505},[213,3180,3181],{"class":368},"}\n",[213,3183,3184,3187,3189,3191,3193,3196,3198,3201],{"class":95,"line":526},[213,3185,3186],{"class":409},"DAILY_BUDGET_USD",[213,3188,413],{"class":364},[213,3190,481],{"class":409},[213,3192,460],{"class":368},[213,3194,3195],{"class":222},"\"DAILY_BUDGET_USD\"",[213,3197,422],{"class":368},[213,3199,3200],{"class":222},"\"25.00\"",[213,3202,471],{"class":368},[213,3204,3205],{"class":95,"line":538},[213,3206,388],{"emptyLinePlaceholder":387},[213,3208,3209],{"class":95,"line":544},[213,3210,388],{"emptyLinePlaceholder":387},[213,3212,3213,3215,3218,3220,3222],{"class":95,"line":553},[213,3214,700],{"class":364},[213,3216,3217],{"class":218}," BudgetExceeded",[213,3219,511],{"class":368},[213,3221,833],{"class":409},[213,3223,711],{"class":368},[213,3225,3226],{"class":95,"line":569},[213,3227,3228],{"class":222},"    \"\"\"Stop the queue rather than discover the overspend on the invoice.\"\"\"\n",[213,3230,3231],{"class":95,"line":585},[213,3232,388],{"emptyLinePlaceholder":387},[213,3234,3235],{"class":95,"line":813},[213,3236,388],{"emptyLinePlaceholder":387},[213,3238,3239,3241,3243,3246,3249,3252,3255,3257,3260,3262,3264,3266],{"class":95,"line":818},[213,3240,887],{"class":364},[213,3242,890],{"class":364},[213,3244,3245],{"class":218}," estimate_usd",[213,3247,3248],{"class":368},"(system: ",[213,3250,3251],{"class":409},"list",[213,3253,3254],{"class":368},", messages: ",[213,3256,3251],{"class":409},[213,3258,3259],{"class":368},", model: ",[213,3261,899],{"class":409},[213,3263,2152],{"class":368},[213,3265,2155],{"class":409},[213,3267,1114],{"class":368},[213,3269,3270,3273,3275,3277],{"class":95,"line":823},[213,3271,3272],{"class":368},"    counted ",[213,3274,518],{"class":364},[213,3276,930],{"class":364},[213,3278,3279],{"class":368}," get_client().messages.count_tokens(\n",[213,3281,3282,3284,3286,3289,3292,3294,3297,3300,3302],{"class":95,"line":838},[213,3283,939],{"class":514},[213,3285,518],{"class":364},[213,3287,3288],{"class":368},"model, ",[213,3290,3291],{"class":514},"system",[213,3293,518],{"class":364},[213,3295,3296],{"class":368},"system, ",[213,3298,3299],{"class":514},"messages",[213,3301,518],{"class":364},[213,3303,3304],{"class":368},"messages\n",[213,3306,3307],{"class":95,"line":844},[213,3308,588],{"class":368},[213,3310,3311,3314,3316,3319],{"class":95,"line":849},[213,3312,3313],{"class":368},"    in_rate, out_rate ",[213,3315,518],{"class":364},[213,3317,3318],{"class":409}," PRICES",[213,3320,3321],{"class":368},"[model]\n",[213,3323,3324,3327,3329,3331,3333,3335,3337,3339,3342,3344],{"class":95,"line":854},[213,3325,3326],{"class":368},"    projected_out ",[213,3328,518],{"class":364},[213,3330,457],{"class":409},[213,3332,460],{"class":368},[213,3334,463],{"class":222},[213,3336,422],{"class":368},[213,3338,468],{"class":222},[213,3340,3341],{"class":368},")) ",[213,3343,1757],{"class":364},[213,3345,3346],{"class":409}," 0.6\n",[213,3348,3349,3351,3354,3356,3359,3362,3365,3367,3370,3373],{"class":95,"line":868},[213,3350,547],{"class":364},[213,3352,3353],{"class":368}," (counted.input_tokens ",[213,3355,1757],{"class":364},[213,3357,3358],{"class":368}," in_rate ",[213,3360,3361],{"class":364},"+",[213,3363,3364],{"class":368}," projected_out ",[213,3366,1757],{"class":364},[213,3368,3369],{"class":368}," out_rate) ",[213,3371,3372],{"class":364},"\u002F",[213,3374,3375],{"class":409}," 1_000_000\n",[213,3377,3378],{"class":95,"line":874},[213,3379,388],{"emptyLinePlaceholder":387},[213,3381,3382],{"class":95,"line":879},[213,3383,388],{"emptyLinePlaceholder":387},[213,3385,3386,3388,3390,3393,3396,3398,3400,3402],{"class":95,"line":884},[213,3387,887],{"class":364},[213,3389,890],{"class":364},[213,3391,3392],{"class":218}," reserve",[213,3394,3395],{"class":368},"(pool: redis.Redis, amount_usd: ",[213,3397,2155],{"class":409},[213,3399,2152],{"class":368},[213,3401,1141],{"class":409},[213,3403,1114],{"class":368},[213,3405,3406,3409,3411,3414,3417,3419,3422,3424],{"class":95,"line":911},[213,3407,3408],{"class":368},"    key ",[213,3410,518],{"class":364},[213,3412,3413],{"class":364}," f",[213,3415,3416],{"class":222},"\"llm:spend:",[213,3418,1032],{"class":409},[213,3420,3421],{"class":368},"date.today().isoformat()",[213,3423,1864],{"class":409},[213,3425,2953],{"class":222},[213,3427,3428,3431,3433,3435],{"class":95,"line":922},[213,3429,3430],{"class":368},"    spent ",[213,3432,518],{"class":364},[213,3434,930],{"class":364},[213,3436,3437],{"class":368}," pool.incrbyfloat(key, amount_usd)\n",[213,3439,3440,3443,3446,3449],{"class":95,"line":936},[213,3441,3442],{"class":364},"    await",[213,3444,3445],{"class":368}," pool.expire(key, ",[213,3447,3448],{"class":409},"172_800",[213,3450,428],{"class":368},[213,3452,3453,3455,3458,3461,3464],{"class":95,"line":949},[213,3454,2842],{"class":364},[213,3456,3457],{"class":368}," spent ",[213,3459,3460],{"class":364},">",[213,3462,3463],{"class":409}," DAILY_BUDGET_USD",[213,3465,1114],{"class":368},[213,3467,3468,3471,3474,3477],{"class":95,"line":961},[213,3469,3470],{"class":364},"        await",[213,3472,3473],{"class":368}," pool.incrbyfloat(key, ",[213,3475,3476],{"class":364},"-",[213,3478,3479],{"class":368},"amount_usd)\n",[213,3481,3482,3484,3487,3489,3491,3493,3496,3499,3501,3504,3507,3509,3511,3513],{"class":95,"line":972},[213,3483,2858],{"class":364},[213,3485,3486],{"class":368}," BudgetExceeded(",[213,3488,1165],{"class":364},[213,3490,1174],{"class":222},[213,3492,1032],{"class":409},[213,3494,3495],{"class":368},"spent",[213,3497,3498],{"class":364},":.2f",[213,3500,1864],{"class":409},[213,3502,3503],{"class":222}," would exceed ",[213,3505,3506],{"class":409},"{DAILY_BUDGET_USD",[213,3508,3498],{"class":364},[213,3510,1864],{"class":409},[213,3512,1174],{"class":222},[213,3514,428],{"class":368},[213,3516,3517],{"class":95,"line":986},[213,3518,388],{"emptyLinePlaceholder":387},[213,3520,3521],{"class":95,"line":995},[213,3522,388],{"emptyLinePlaceholder":387},[213,3524,3525,3527,3530,3533,3535,3538,3541,3543,3545],{"class":95,"line":1018},[213,3526,529],{"class":364},[213,3528,3529],{"class":218}," choose_model",[213,3531,3532],{"class":368},"(input_tokens: ",[213,3534,734],{"class":409},[213,3536,3537],{"class":368},", needs_reasoning: ",[213,3539,3540],{"class":409},"bool",[213,3542,2152],{"class":368},[213,3544,899],{"class":409},[213,3546,1114],{"class":368},[213,3548,3549],{"class":95,"line":1024},[213,3550,3551],{"class":222},"    \"\"\"Route cheap, mechanical work to the small model. Measure before trusting it.\"\"\"\n",[213,3553,3554,3556,3558,3561,3564,3566,3569,3571,3574,3577],{"class":95,"line":1059},[213,3555,547],{"class":364},[213,3557,668],{"class":409},[213,3559,3560],{"class":364}," if",[213,3562,3563],{"class":368}," needs_reasoning ",[213,3565,2809],{"class":364},[213,3567,3568],{"class":368}," input_tokens ",[213,3570,3460],{"class":364},[213,3572,3573],{"class":409}," 20_000",[213,3575,3576],{"class":364}," else",[213,3578,3579],{"class":409}," CHEAP_MODEL\n",[14,3581,3582,3583,3586,3587,3591],{},"Counting tokens before you send is the whole game. ",[187,3584,3585],{},"count_tokens"," is a cheap call that tells you what a job will cost before you commit to it, which means the queue can reject a 400,000-token document rather than silently spending four dollars on one customer's PDF dump. Pair the reservation counter with the same Redis instance you use for ",[18,3588,3590],{"href":3589},"\u002Fscaling-and-operating-production-python-apis\u002Fcaching-python-api-responses-with-redis\u002F","caching API responses with Redis"," — the budget ledger is a few keys, not a new dependency.",[14,3593,3594],{},"For the patient half of the workload, submit a batch and poll:",[204,3596,3598],{"className":349,"code":3597,"language":351,"meta":209,"style":209},"# llm\u002Foffline.py\nimport asyncio\nimport os\n\nfrom .client import MODEL, MAX_TOKENS, get_client\nfrom .prompts import load_prompt\n\nPOLL_SECONDS = int(os.getenv(\"BATCH_POLL_SECONDS\", \"60\"))\n\n\nasync def submit_offline(jobs: dict[str, str]) -> str:\n    system_text, _ = load_prompt()\n    batch = await get_client().messages.batches.create(\n        requests=[\n            {\n                \"custom_id\": job_id,\n                \"params\": {\n                    \"model\": MODEL,\n                    \"max_tokens\": MAX_TOKENS,\n                    \"system\": [{\"type\": \"text\", \"text\": system_text}],\n                    \"messages\": [{\"role\": \"user\", \"content\": raw}],\n                },\n            }\n            for job_id, raw in jobs.items()\n        ]\n    )\n    return batch.id\n\n\nasync def collect_offline(batch_id: str) -> dict[str, str]:\n    client = get_client()\n    while True:\n        batch = await client.messages.batches.retrieve(batch_id)\n        if batch.processing_status == \"ended\":\n            break\n        await asyncio.sleep(POLL_SECONDS)\n\n    out: dict[str, str] = {}\n    async for entry in await client.messages.batches.results(batch_id):\n        if entry.result.type == \"succeeded\":\n            blocks = entry.result.message.content\n            out[entry.custom_id] = \"\".join(b.text for b in blocks if b.type == \"text\")\n    return out\n",[187,3599,3600,3605,3611,3617,3621,3637,3647,3651,3672,3676,3680,3704,3713,3725,3735,3740,3748,3756,3767,3778,3799,3818,3823,3828,3841,3846,3850,3857,3861,3865,3890,3900,3910,3922,3937,3942,3953,3957,3975,3991,4005,4015,4045],{"__ignoreMap":209},[213,3601,3602],{"class":95,"line":215},[213,3603,3604],{"class":358},"# llm\u002Foffline.py\n",[213,3606,3607,3609],{"class":95,"line":264},[213,3608,365],{"class":364},[213,3610,1441],{"class":368},[213,3612,3613,3615],{"class":95,"line":270},[213,3614,365],{"class":364},[213,3616,369],{"class":368},[213,3618,3619],{"class":95,"line":276},[213,3620,388],{"emptyLinePlaceholder":387},[213,3622,3623,3625,3627,3629,3631,3633,3635],{"class":95,"line":282},[213,3624,374],{"class":364},[213,3626,663],{"class":368},[213,3628,365],{"class":364},[213,3630,668],{"class":409},[213,3632,422],{"class":368},[213,3634,452],{"class":409},[213,3636,675],{"class":368},[213,3638,3639,3641,3643,3645],{"class":95,"line":288},[213,3640,374],{"class":364},[213,3642,682],{"class":368},[213,3644,365],{"class":364},[213,3646,687],{"class":368},[213,3648,3649],{"class":95,"line":294},[213,3650,388],{"emptyLinePlaceholder":387},[213,3652,3653,3656,3658,3660,3662,3665,3667,3670],{"class":95,"line":300},[213,3654,3655],{"class":409},"POLL_SECONDS",[213,3657,413],{"class":364},[213,3659,457],{"class":409},[213,3661,460],{"class":368},[213,3663,3664],{"class":222},"\"BATCH_POLL_SECONDS\"",[213,3666,422],{"class":368},[213,3668,3669],{"class":222},"\"60\"",[213,3671,471],{"class":368},[213,3673,3674],{"class":95,"line":306},[213,3675,388],{"emptyLinePlaceholder":387},[213,3677,3678],{"class":95,"line":312},[213,3679,388],{"emptyLinePlaceholder":387},[213,3681,3682,3684,3686,3689,3691,3693,3695,3697,3700,3702],{"class":95,"line":318},[213,3683,887],{"class":364},[213,3685,890],{"class":364},[213,3687,3688],{"class":218}," submit_offline",[213,3690,1699],{"class":368},[213,3692,899],{"class":409},[213,3694,422],{"class":368},[213,3696,899],{"class":409},[213,3698,3699],{"class":368},"]) -> ",[213,3701,899],{"class":409},[213,3703,1114],{"class":368},[213,3705,3706,3709,3711],{"class":95,"line":500},[213,3707,3708],{"class":368},"    system_text, _ ",[213,3710,518],{"class":364},[213,3712,919],{"class":368},[213,3714,3715,3718,3720,3722],{"class":95,"line":505},[213,3716,3717],{"class":368},"    batch ",[213,3719,518],{"class":364},[213,3721,930],{"class":364},[213,3723,3724],{"class":368}," get_client().messages.batches.create(\n",[213,3726,3727,3730,3732],{"class":95,"line":526},[213,3728,3729],{"class":514},"        requests",[213,3731,518],{"class":364},[213,3733,3734],{"class":368},"[\n",[213,3736,3737],{"class":95,"line":538},[213,3738,3739],{"class":368},"            {\n",[213,3741,3742,3745],{"class":95,"line":544},[213,3743,3744],{"class":222},"                \"custom_id\"",[213,3746,3747],{"class":368},": job_id,\n",[213,3749,3750,3753],{"class":95,"line":553},[213,3751,3752],{"class":222},"                \"params\"",[213,3754,3755],{"class":368},": {\n",[213,3757,3758,3761,3763,3765],{"class":95,"line":569},[213,3759,3760],{"class":222},"                    \"model\"",[213,3762,978],{"class":368},[213,3764,410],{"class":409},[213,3766,946],{"class":368},[213,3768,3769,3772,3774,3776],{"class":95,"line":585},[213,3770,3771],{"class":222},"                    \"max_tokens\"",[213,3773,978],{"class":368},[213,3775,452],{"class":409},[213,3777,946],{"class":368},[213,3779,3780,3783,3786,3788,3790,3792,3794,3796],{"class":95,"line":813},[213,3781,3782],{"class":222},"                    \"system\"",[213,3784,3785],{"class":368},": [{",[213,3787,1004],{"class":222},[213,3789,978],{"class":368},[213,3791,981],{"class":222},[213,3793,422],{"class":368},[213,3795,981],{"class":222},[213,3797,3798],{"class":368},": system_text}],\n",[213,3800,3801,3804,3806,3808,3810,3812,3814,3816],{"class":95,"line":818},[213,3802,3803],{"class":222},"                    \"messages\"",[213,3805,3785],{"class":368},[213,3807,1070],{"class":222},[213,3809,978],{"class":368},[213,3811,1075],{"class":222},[213,3813,422],{"class":368},[213,3815,1080],{"class":222},[213,3817,1083],{"class":368},[213,3819,3820],{"class":95,"line":823},[213,3821,3822],{"class":368},"                },\n",[213,3824,3825],{"class":95,"line":838},[213,3826,3827],{"class":368},"            }\n",[213,3829,3830,3833,3836,3838],{"class":95,"line":844},[213,3831,3832],{"class":364},"            for",[213,3834,3835],{"class":368}," job_id, raw ",[213,3837,1204],{"class":364},[213,3839,3840],{"class":368}," jobs.items()\n",[213,3842,3843],{"class":95,"line":849},[213,3844,3845],{"class":368},"        ]\n",[213,3847,3848],{"class":95,"line":854},[213,3849,588],{"class":368},[213,3851,3852,3854],{"class":95,"line":868},[213,3853,547],{"class":364},[213,3855,3856],{"class":368}," batch.id\n",[213,3858,3859],{"class":95,"line":874},[213,3860,388],{"emptyLinePlaceholder":387},[213,3862,3863],{"class":95,"line":879},[213,3864,388],{"emptyLinePlaceholder":387},[213,3866,3867,3869,3871,3874,3877,3879,3882,3884,3886,3888],{"class":95,"line":884},[213,3868,887],{"class":364},[213,3870,890],{"class":364},[213,3872,3873],{"class":218}," collect_offline",[213,3875,3876],{"class":368},"(batch_id: ",[213,3878,899],{"class":409},[213,3880,3881],{"class":368},") -> dict[",[213,3883,899],{"class":409},[213,3885,422],{"class":368},[213,3887,899],{"class":409},[213,3889,908],{"class":368},[213,3891,3892,3895,3897],{"class":95,"line":911},[213,3893,3894],{"class":368},"    client ",[213,3896,518],{"class":364},[213,3898,3899],{"class":368}," get_client()\n",[213,3901,3902,3905,3908],{"class":95,"line":922},[213,3903,3904],{"class":364},"    while",[213,3906,3907],{"class":409}," True",[213,3909,1114],{"class":368},[213,3911,3912,3915,3917,3919],{"class":95,"line":936},[213,3913,3914],{"class":368},"        batch ",[213,3916,518],{"class":364},[213,3918,930],{"class":364},[213,3920,3921],{"class":368}," client.messages.batches.retrieve(batch_id)\n",[213,3923,3924,3927,3930,3932,3935],{"class":95,"line":949},[213,3925,3926],{"class":364},"        if",[213,3928,3929],{"class":368}," batch.processing_status ",[213,3931,1216],{"class":364},[213,3933,3934],{"class":222}," \"ended\"",[213,3936,1114],{"class":368},[213,3938,3939],{"class":95,"line":961},[213,3940,3941],{"class":364},"            break\n",[213,3943,3944,3946,3949,3951],{"class":95,"line":972},[213,3945,3470],{"class":364},[213,3947,3948],{"class":368}," asyncio.sleep(",[213,3950,3655],{"class":409},[213,3952,428],{"class":368},[213,3954,3955],{"class":95,"line":986},[213,3956,388],{"emptyLinePlaceholder":387},[213,3958,3959,3962,3964,3966,3968,3970,3972],{"class":95,"line":995},[213,3960,3961],{"class":368},"    out: dict[",[213,3963,899],{"class":409},[213,3965,422],{"class":368},[213,3967,899],{"class":409},[213,3969,802],{"class":368},[213,3971,518],{"class":364},[213,3973,3974],{"class":368}," {}\n",[213,3976,3977,3979,3981,3984,3986,3988],{"class":95,"line":1018},[213,3978,1620],{"class":364},[213,3980,2290],{"class":364},[213,3982,3983],{"class":368}," entry ",[213,3985,1204],{"class":364},[213,3987,930],{"class":364},[213,3989,3990],{"class":368}," client.messages.batches.results(batch_id):\n",[213,3992,3993,3995,3998,4000,4003],{"class":95,"line":1024},[213,3994,3926],{"class":364},[213,3996,3997],{"class":368}," entry.result.type ",[213,3999,1216],{"class":364},[213,4001,4002],{"class":222}," \"succeeded\"",[213,4004,1114],{"class":368},[213,4006,4007,4010,4012],{"class":95,"line":1059},[213,4008,4009],{"class":368},"            blocks ",[213,4011,518],{"class":364},[213,4013,4014],{"class":368}," entry.result.message.content\n",[213,4016,4017,4020,4022,4024,4026,4028,4030,4032,4035,4037,4039,4041,4043],{"class":95,"line":1086},[213,4018,4019],{"class":368},"            out[entry.custom_id] ",[213,4021,518],{"class":364},[213,4023,1192],{"class":222},[213,4025,1195],{"class":368},[213,4027,1198],{"class":364},[213,4029,1201],{"class":368},[213,4031,1204],{"class":364},[213,4033,4034],{"class":368}," blocks ",[213,4036,1210],{"class":364},[213,4038,1213],{"class":368},[213,4040,1216],{"class":364},[213,4042,1219],{"class":222},[213,4044,428],{"class":368},[213,4046,4047,4049],{"class":95,"line":1091},[213,4048,547],{"class":364},[213,4050,1958],{"class":368},[14,4052,4053,4054,4057,4058,4062],{},"Results come back in arbitrary order, so key them by ",[187,4055,4056],{},"custom_id"," and never by position. A batch usually finishes inside an hour and is guaranteed within twenty-four, which makes it perfect for anything driven by ",[18,4059,4061],{"href":4060},"\u002Fautomating-side-hustle-operations-with-apis\u002Fscheduling-data-pipelines-with-cron\u002F","scheduled data pipelines"," and useless for anything a human is waiting on.",[176,4064],{},[179,4066,4068],{"id":4067},"configuration-reference","Configuration reference",[4070,4071,4072,4088],"table",{},[4073,4074,4075],"thead",{},[4076,4077,4078,4082,4085],"tr",{},[4079,4080,4081],"th",{},"Env var",[4079,4083,4084],{},"Default",[4079,4086,4087],{},"Production recommendation",[4089,4090,4091,4107,4122,4137,4152,4166,4181,4195,4209,4222],"tbody",{},[4076,4092,4093,4099,4104],{},[4094,4095,4096],"td",{},[187,4097,4098],{},"LLM_MODEL",[4094,4100,4101],{},[187,4102,4103],{},"claude-opus-4-8",[4094,4105,4106],{},"Pin explicitly; never float",[4076,4108,4109,4114,4119],{},[4094,4110,4111],{},[187,4112,4113],{},"LLM_CHEAP_MODEL",[4094,4115,4116],{},[187,4117,4118],{},"claude-haiku-4-5",[4094,4120,4121],{},"Route mechanical jobs here",[4076,4123,4124,4129,4134],{},[4094,4125,4126],{},[187,4127,4128],{},"LLM_MAX_TOKENS",[4094,4130,4131],{},[187,4132,4133],{},"2048",[4094,4135,4136],{},"1.5x your longest valid output",[4076,4138,4139,4144,4149],{},[4094,4140,4141],{},[187,4142,4143],{},"LLM_REQUEST_TIMEOUT_S",[4094,4145,4146],{},[187,4147,4148],{},"90",[4094,4150,4151],{},"Measured p99 plus 30s",[4076,4153,4154,4159,4163],{},[4094,4155,4156],{},[187,4157,4158],{},"LLM_CONCURRENCY",[4094,4160,4161],{},[187,4162,2758],{},[4094,4164,4165],{},"25% of your token-rate ceiling",[4076,4167,4168,4173,4178],{},[4094,4169,4170],{},[187,4171,4172],{},"LLM_MAX_ATTEMPTS",[4094,4174,4175],{},[187,4176,4177],{},"4",[4094,4179,4180],{},"4 for queued work, 2 for live",[4076,4182,4183,4187,4192],{},[4094,4184,4185],{},[187,4186,2703],{},[4094,4188,4189],{},[187,4190,4191],{},"prompts\u002F*.toml",[4094,4193,4194],{},"Ship in the image, not a volume",[4076,4196,4197,4202,4206],{},[4094,4198,4199],{},[187,4200,4201],{},"PROMPT_VERSION",[4094,4203,4204],{},[187,4205,2605],{},[4094,4207,4208],{},"The rollback switch — keep it",[4076,4210,4211,4215,4219],{},[4094,4212,4213],{},[187,4214,3186],{},[4094,4216,4217],{},[187,4218,3158],{},[4094,4220,4221],{},"Set to 1.3x expected daily spend",[4076,4223,4224,4229,4233],{},[4094,4225,4226],{},[187,4227,4228],{},"BATCH_POLL_SECONDS",[4094,4230,4231],{},[187,4232,84],{},[4094,4234,4235],{},"60s; batches take minutes to hours",[14,4237,4238,4239,4241,4242,4244],{},"Two of these deserve emphasis. ",[187,4240,4128],{}," is not a safety limit — it is a truncation point, and setting it too low produces invalid JSON rather than a friendly error. ",[187,4243,3186],{}," is the only line of defence between a runaway retry loop and a four-figure invoice, so it belongs in the environment of every deployment, including staging.",[176,4246],{},[179,4248,4250],{"id":4249},"gotchas-and-failure-modes","Gotchas and failure modes",[14,4252,4253,4256,4257,4260,4261,4264,4265,4268,4269,1975],{},[343,4254,4255],{},"A synchronous client inside an async worker."," Symptom: you set ",[187,4258,4259],{},"LLM_CONCURRENCY=16"," and throughput stays flat at one job at a time, with CPU near zero. Cause: someone imported the synchronous ",[187,4262,4263],{},"Anthropic"," client, so every call blocks the event loop for fifty seconds while fifteen other coroutines wait their turn. Fix: use ",[187,4266,4267],{},"AsyncAnthropic"," everywhere and grep the codebase for the sync class. If a legacy sync call must stay, wrap it in ",[187,4270,4271],{},"asyncio.to_thread",[14,4273,4274,4277,4278,4281,4282,4285,4286,4289,4290,4292],{},[343,4275,4276],{},"Truncated JSON masquerading as a model quality problem."," Symptom: ",[187,4279,4280],{},"ValidationError"," spikes on long input documents while short ones extract perfectly. Cause: the response hit ",[187,4283,4284],{},"max_tokens"," mid-object; the schema constraint guarantees the format of what was emitted, not that emission finished. Fix: check ",[187,4287,4288],{},"stop_reason == \"max_tokens\""," explicitly before parsing, as Step 2 does, and raise ",[187,4291,4128],{},". Retrying the identical request just truncates again.",[14,4294,4295,4277,4298,4300],{},[343,4296,4297],{},"Prompt caching that never hits.",[187,4299,3055],{}," is zero on every response and your bill matches the uncached estimate exactly. Cause: something volatile sits in the prefix — a timestamp, a job ID, a customer name interpolated into the system prompt — or the prefix is below the model's minimum cacheable size. Fix: freeze the system block, move all per-job context into the user message, and verify with a real response rather than assuming.",[14,4302,4303,4306,4307,4310,4311,4314],{},[343,4304,4305],{},"Retrying refusals and 400s."," Symptom: a small number of jobs each consume four attempts and appear in the dead-letter queue anyway, and your cost per successful job is 15% higher than the arithmetic says it should be. Cause: a blanket ",[187,4308,4309],{},"except Exception"," around the retry wrapper. Fix: ",[187,4312,4313],{},"retry_if_exception_type(TRANSIENT)"," with an explicit tuple, as in Step 4. A refusal is an answer, not an error.",[14,4316,4317,4320,4321,4324,4325,4328,4329,4332,4333,4337],{},[343,4318,4319],{},"Thread-unsafe budget accounting."," Symptom: the daily spend counter drifts below reality and the cap never triggers. Cause: read-modify-write against Redis from multiple workers. Fix: ",[187,4322,4323],{},"INCRBYFLOAT"," is atomic — reserve first, refund on failure, never ",[187,4326,4327],{},"GET"," then ",[187,4330,4331],{},"SET",". The same reasoning behind ",[18,4334,4336],{"href":4335},"\u002Fautomating-side-hustle-operations-with-apis\u002Fprocessing-webhooks-with-python\u002Fbuilding-an-idempotent-webhook-receiver\u002F","building an idempotent webhook receiver"," applies to spend: assume every operation runs twice.",[176,4339],{},[179,4341,4343],{"id":4342},"verification","Verification",[14,4345,4346],{},"Prove three things before you call this done: that the schema constraint holds, that the budget gate fires, and that a real call produces the token counts you expect.",[14,4348,4349],{},"Start with a live smoke test that prints exactly what you are being billed for:",[204,4351,4353],{"className":349,"code":4352,"language":351,"meta":209,"style":209},"# scripts\u002Fsmoke.py\nimport asyncio\n\nfrom llm.extract import extract_lead\n\nSAMPLE = \"Hi - we're Northwind Ltd, budget is 4-6k, can we see a demo? tom@northwind.example\"\n\n\nasync def main() -> None:\n    lead, meta = await extract_lead(SAMPLE)\n    print(lead.model_dump_json(indent=2))\n    print(meta)\n\n\nasyncio.run(main())\n",[187,4354,4355,4360,4366,4370,4382,4386,4396,4400,4404,4420,4436,4453,4460,4464,4468],{"__ignoreMap":209},[213,4356,4357],{"class":95,"line":215},[213,4358,4359],{"class":358},"# scripts\u002Fsmoke.py\n",[213,4361,4362,4364],{"class":95,"line":264},[213,4363,365],{"class":364},[213,4365,1441],{"class":368},[213,4367,4368],{"class":95,"line":270},[213,4369,388],{"emptyLinePlaceholder":387},[213,4371,4372,4374,4377,4379],{"class":95,"line":276},[213,4373,374],{"class":364},[213,4375,4376],{"class":368}," llm.extract ",[213,4378,365],{"class":364},[213,4380,4381],{"class":368}," extract_lead\n",[213,4383,4384],{"class":95,"line":282},[213,4385,388],{"emptyLinePlaceholder":387},[213,4387,4388,4391,4393],{"class":95,"line":288},[213,4389,4390],{"class":409},"SAMPLE",[213,4392,413],{"class":364},[213,4394,4395],{"class":222}," \"Hi - we're Northwind Ltd, budget is 4-6k, can we see a demo? tom@northwind.example\"\n",[213,4397,4398],{"class":95,"line":294},[213,4399,388],{"emptyLinePlaceholder":387},[213,4401,4402],{"class":95,"line":300},[213,4403,388],{"emptyLinePlaceholder":387},[213,4405,4406,4408,4410,4413,4416,4418],{"class":95,"line":306},[213,4407,887],{"class":364},[213,4409,890],{"class":364},[213,4411,4412],{"class":218}," main",[213,4414,4415],{"class":368},"() -> ",[213,4417,1141],{"class":409},[213,4419,1114],{"class":368},[213,4421,4422,4425,4427,4429,4432,4434],{"class":95,"line":312},[213,4423,4424],{"class":368},"    lead, meta ",[213,4426,518],{"class":364},[213,4428,930],{"class":364},[213,4430,4431],{"class":368}," extract_lead(",[213,4433,4390],{"class":409},[213,4435,428],{"class":368},[213,4437,4438,4441,4444,4447,4449,4451],{"class":95,"line":318},[213,4439,4440],{"class":409},"    print",[213,4442,4443],{"class":368},"(lead.model_dump_json(",[213,4445,4446],{"class":514},"indent",[213,4448,518],{"class":364},[213,4450,2364],{"class":409},[213,4452,471],{"class":368},[213,4454,4455,4457],{"class":95,"line":500},[213,4456,4440],{"class":409},[213,4458,4459],{"class":368},"(meta)\n",[213,4461,4462],{"class":95,"line":505},[213,4463,388],{"emptyLinePlaceholder":387},[213,4465,4466],{"class":95,"line":526},[213,4467,388],{"emptyLinePlaceholder":387},[213,4469,4470],{"class":95,"line":538},[213,4471,4472],{"class":368},"asyncio.run(main())\n",[14,4474,4475,4476,4479,4480,4483,4484,4487],{},"Run it with ",[187,4477,4478],{},"python -m scripts.smoke",". Expected output is a valid ",[187,4481,4482],{},"Lead"," object plus a metadata dict whose ",[187,4485,4486],{},"cache_read_tokens"," is zero on the first run and non-zero on the second — that second run is the only proof that caching is working:",[204,4489,4493],{"className":4490,"code":4492,"language":72,"meta":209},[4491],"language-text","{\"company\": \"Northwind Ltd\", \"contact_email\": \"tom@northwind.example\",\n \"budget_usd\": 4000, \"intent\": \"demo\"}\n{'prompt_digest': 'v3:e30f57a91c4d', 'input_tokens': 118,\n 'output_tokens': 74, 'cache_read_tokens': 5211}\n",[187,4494,4492],{"__ignoreMap":209},[14,4496,4497,4498,4502],{},"Then lock the budget gate down in a test that never touches the network. Mocking the model client rather than the HTTP layer keeps the test fast; the alternative approach, intercepting requests with ",[18,4499,4501],{"href":4500},"\u002Fscaling-and-operating-production-python-apis\u002Ftesting-python-apis-with-pytest\u002Fmocking-external-apis-with-respx\u002F","respx",", is better when you want to assert on the wire format itself:",[204,4504,4506],{"className":349,"code":4505,"language":351,"meta":209,"style":209},"# tests\u002Ftest_budget.py\nimport pytest\nimport redis.asyncio as redis\n\nfrom llm.budget import BudgetExceeded, reserve\n\n\n@pytest.mark.asyncio\nasync def test_reserve_blocks_over_budget(monkeypatch):\n    monkeypatch.setenv(\"DAILY_BUDGET_USD\", \"1.00\")\n    pool = redis.from_url(\"redis:\u002F\u002Flocalhost:6379\u002F15\", decode_responses=True)\n    await pool.flushdb()\n\n    await reserve(pool, 0.90)\n    with pytest.raises(BudgetExceeded):\n        await reserve(pool, 0.50)\n\n    key_count = await pool.dbsize()\n    assert key_count == 1          # the refund left no orphan keys\n",[187,4507,4508,4513,4520,4530,4534,4546,4550,4554,4559,4571,4585,4609,4616,4620,4632,4640,4651,4655,4667],{"__ignoreMap":209},[213,4509,4510],{"class":95,"line":215},[213,4511,4512],{"class":358},"# tests\u002Ftest_budget.py\n",[213,4514,4515,4517],{"class":95,"line":264},[213,4516,365],{"class":364},[213,4518,4519],{"class":368}," pytest\n",[213,4521,4522,4524,4526,4528],{"class":95,"line":270},[213,4523,365],{"class":364},[213,4525,3098],{"class":368},[213,4527,1839],{"class":364},[213,4529,3103],{"class":368},[213,4531,4532],{"class":95,"line":276},[213,4533,388],{"emptyLinePlaceholder":387},[213,4535,4536,4538,4541,4543],{"class":95,"line":282},[213,4537,374],{"class":364},[213,4539,4540],{"class":368}," llm.budget ",[213,4542,365],{"class":364},[213,4544,4545],{"class":368}," BudgetExceeded, reserve\n",[213,4547,4548],{"class":95,"line":288},[213,4549,388],{"emptyLinePlaceholder":387},[213,4551,4552],{"class":95,"line":294},[213,4553,388],{"emptyLinePlaceholder":387},[213,4555,4556],{"class":95,"line":300},[213,4557,4558],{"class":218},"@pytest.mark.asyncio\n",[213,4560,4561,4563,4565,4568],{"class":95,"line":306},[213,4562,887],{"class":364},[213,4564,890],{"class":364},[213,4566,4567],{"class":218}," test_reserve_blocks_over_budget",[213,4569,4570],{"class":368},"(monkeypatch):\n",[213,4572,4573,4576,4578,4580,4583],{"class":95,"line":312},[213,4574,4575],{"class":368},"    monkeypatch.setenv(",[213,4577,3195],{"class":222},[213,4579,422],{"class":368},[213,4581,4582],{"class":222},"\"1.00\"",[213,4584,428],{"class":368},[213,4586,4587,4590,4592,4595,4598,4600,4603,4605,4607],{"class":95,"line":318},[213,4588,4589],{"class":368},"    pool ",[213,4591,518],{"class":364},[213,4593,4594],{"class":368}," redis.from_url(",[213,4596,4597],{"class":222},"\"redis:\u002F\u002Flocalhost:6379\u002F15\"",[213,4599,422],{"class":368},[213,4601,4602],{"class":514},"decode_responses",[213,4604,518],{"class":364},[213,4606,1538],{"class":409},[213,4608,428],{"class":368},[213,4610,4611,4613],{"class":95,"line":500},[213,4612,3442],{"class":364},[213,4614,4615],{"class":368}," pool.flushdb()\n",[213,4617,4618],{"class":95,"line":505},[213,4619,388],{"emptyLinePlaceholder":387},[213,4621,4622,4624,4627,4630],{"class":95,"line":526},[213,4623,3442],{"class":364},[213,4625,4626],{"class":368}," reserve(pool, ",[213,4628,4629],{"class":409},"0.90",[213,4631,428],{"class":368},[213,4633,4634,4637],{"class":95,"line":538},[213,4635,4636],{"class":364},"    with",[213,4638,4639],{"class":368}," pytest.raises(BudgetExceeded):\n",[213,4641,4642,4644,4646,4649],{"class":95,"line":544},[213,4643,3470],{"class":364},[213,4645,4626],{"class":368},[213,4647,4648],{"class":409},"0.50",[213,4650,428],{"class":368},[213,4652,4653],{"class":95,"line":553},[213,4654,388],{"emptyLinePlaceholder":387},[213,4656,4657,4660,4662,4664],{"class":95,"line":569},[213,4658,4659],{"class":368},"    key_count ",[213,4661,518],{"class":364},[213,4663,930],{"class":364},[213,4665,4666],{"class":368}," pool.dbsize()\n",[213,4668,4669,4672,4675,4677,4680],{"class":95,"line":585},[213,4670,4671],{"class":364},"    assert",[213,4673,4674],{"class":368}," key_count ",[213,4676,1216],{"class":364},[213,4678,4679],{"class":409}," 1",[213,4681,4682],{"class":358},"          # the refund left no orphan keys\n",[14,4684,4685,4686,422,4688,422,4691,422,4694,422,4696,4699,4700,4703,4704,4708],{},"Finally, emit one structured log line per job carrying ",[187,4687,2959],{},[187,4689,4690],{},"input_tokens",[187,4692,4693],{},"output_tokens",[187,4695,4486],{},[187,4697,4698],{},"attempt",", and ",[187,4701,4702],{},"usd",". That single line is your entire AI observability story; ",[18,4705,4707],{"href":4706},"\u002Fscaling-and-operating-production-python-apis\u002Fmonitoring-and-logging-python-apis\u002Fstructured-logging-with-structlog\u002F","structured logging with structlog"," covers wiring it into a JSON handler.",[176,4710],{},[179,4712,4714],{"id":4713},"cost-and-performance-at-scale","Cost and performance at scale",[14,4716,4717,4718,4721,4722,4724,4725,4728,4729,4732,4733,4736,4737,1975],{},"Here is the arithmetic that decides whether this pipeline is a business or a hobby. Take a realistic extraction job: a 5,000-token stable prefix (system prompt, schema, examples), a 1,000-token customer document, and 700 tokens of structured output. On a frontier model at $5 per million input tokens and $25 per million output tokens, the naive version costs 6,000 × $5 ÷ 1M for input plus 700 × $25 ÷ 1M for output — three cents plus 1.75 cents, or ",[343,4719,4720],{},"$47.50 per thousand jobs",". Add ",[187,4723,1295],{}," to the prefix and those 5,000 shared tokens bill at roughly a tenth of the input rate, dropping input cost from $0.030 to $0.0075 and the total to ",[343,4726,4727],{},"$25.00 per thousand",". Push the patient half through the Batch API at 50% off and it lands at ",[343,4730,4731],{},"$12.50 per thousand",". Route the mechanical jobs to a small model at $1 and $5 per million and you are at ",[343,4734,4735],{},"$5.00 per thousand"," — a tenth of where you started, with the same code and the same prompt. Throughput follows the same shape: at eight concurrent workers and fifty seconds per call, one process sustains roughly 576 jobs per hour, so a single small container clears a 10,000-job backlog overnight. The number to instrument is cost per successful job, not cost per call — retries, refusals, and validation failures all inflate the denominator, and the difference between the two is usually 10 to 20%. The method for turning that into a price is in ",[18,4738,4740],{"href":4739},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fcalculating-cost-per-api-request\u002F","calculating cost per API request",[27,4742,36,4747,36,4750,36,4753,36,4756,36,4759,36,4763,36,4767,36,4770,36,4773,36,4779,36,4783,36,4787,36,4791,36,4794,36,4797,36,4801,36,4804,36,4807,36,4810,36,4815,36,4818,36,4822,36,4824,36,4827,36,4829,36,4832,36,4835,36,4837],{"viewBox":4743,"role":30,"ariaLabelledBy":4744,"xmlns":34,"style":35},"0 0 720 320",[4745,4746],"aiwf-cost-t","aiwf-cost-d",[38,4748,4749],{"id":4745},"Cost per thousand extraction jobs under four configurations",[42,4751,4752],{"id":4746},"Bar chart comparing cost per thousand jobs: forty-seven dollars fifty with no caching, twenty-five dollars with prompt caching, twelve dollars fifty adding batch submission, and five dollars on a small model with caching.",[46,4754],{"x":48,"y":48,"width":1311,"height":4755,"fill":51},"320",[72,4757,4758],{"x":1322,"y":1323,"fill":76,"style":77},"Cost per 1,000 extraction jobs",[72,4760,4762],{"x":1322,"y":4761,"fill":70,"style":137},"46","5,000-token shared prefix, 1,000-token document, 700-token output",[95,4764],{"x1":4765,"y1":2975,"x2":4766,"y2":2975,"stroke":150,"style":87},"50","690",[95,4768],{"x1":4765,"y1":1328,"x2":4766,"y2":1328,"stroke":150,"style":4769},"stroke-width:1;stroke-dasharray:4 4;",[95,4771],{"x1":4765,"y1":4772,"x2":4766,"y2":4772,"stroke":150,"style":4769},"70",[72,4774,4778],{"x":4775,"y":4776,"fill":70,"style":4777},"42","254","text-anchor:end;font-size:11;font-family:var(--font-sans);","$0",[72,4780,4782],{"x":4775,"y":4781,"fill":70,"style":4777},"164","$25",[72,4784,4786],{"x":4775,"y":4785,"fill":70,"style":4777},"74","$50",[46,4788],{"x":4148,"y":4789,"width":91,"height":4790,"rx":4177,"fill":116},"79","171",[72,4792,4793],{"x":2986,"y":4772,"fill":76,"style":173},"$47.50",[46,4795],{"x":4796,"y":1328,"width":91,"height":4148,"rx":4177,"fill":1372},"240",[72,4798,4800],{"x":50,"y":4799,"fill":76,"style":173},"151","$25.00",[46,4802],{"x":115,"y":103,"width":91,"height":4803,"rx":4177,"fill":86},"45",[72,4805,4806],{"x":1349,"y":2471,"fill":76,"style":173},"$12.50",[46,4808],{"x":123,"y":1389,"width":91,"height":4809,"rx":4177,"fill":104},"18",[72,4811,4814],{"x":4812,"y":4813,"fill":76,"style":173},"590","223","$5.00",[72,4816,4817],{"x":2986,"y":2498,"fill":76,"style":137},"Frontier",[72,4819,4821],{"x":2986,"y":4820,"fill":70,"style":137},"288","no cache",[72,4823,4817],{"x":50,"y":2498,"fill":76,"style":137},[72,4825,4826],{"x":50,"y":4820,"fill":70,"style":137},"+cache",[72,4828,4817],{"x":1349,"y":2498,"fill":76,"style":137},[72,4830,4831],{"x":1349,"y":4820,"fill":70,"style":137},"+cache +batch",[72,4833,4834],{"x":4812,"y":2498,"fill":76,"style":137},"Small model",[72,4836,4826],{"x":4812,"y":4820,"fill":70,"style":137},[72,4838,4840],{"x":1322,"y":4839,"fill":70,"style":137},"312","Same code, same prompt — only configuration differs",[176,4842],{},[179,4844,4846],{"id":4845},"faq","FAQ",[14,4848,4849,4852],{},[343,4850,4851],{},"What does an AI extraction pipeline actually cost to run at 10,000 jobs a month?","\nUsing the worked example above — a shared 5,000-token prefix, a 1,000-token document, and 700 tokens out — you land at roughly $250 a month on a frontier model with prompt caching, $125 if the work can go through the batch endpoint, and about $50 if the jobs are mechanical enough for a small model. Compute is noise next to that: a single small container handles the whole volume. Budget 15% on top for retries and dead-lettered jobs, and instrument cost per successful job so you notice when that overhead grows.",[14,4854,4855,4858,4859,4863],{},[343,4856,4857],{},"Should I resell this as a per-seat feature or meter it per job?","\nMeter it, at least until your usage distribution stops surprising you. AI cost scales with input length, which scales with how much your heaviest customer pastes in, and a seat price averages that away in exactly the wrong direction — your worst-margin customer is also your happiest one. Start usage-based, cap the free tier by token volume rather than job count, and revisit once you have three months of data. The ",[18,4860,4862],{"href":4861},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fusage-based-vs-seat-based-pricing\u002F","usage-based versus seat-based pricing"," trade-offs go deeper.",[14,4865,4866,4869,4870,4872],{},[343,4867,4868],{},"How risky is it to switch model providers later?","\nLower than it feels, if you keep three seams clean: the client construction, the prompt registry, and the schema. Those are the only three places provider specifics appear in the code above; the queueing, retry, budget, and validation layers are provider-agnostic. What does not port is calibration — your prompt version, your ",[187,4871,4284],{}," value, and your quality baseline all need re-measuring, which is a week of eval work rather than a rewrite. Keep an eval set of a few hundred labelled examples and the migration becomes a number rather than an argument.",[14,4874,4875,4878,4879,4883,4884,4888],{},[343,4876,4877],{},"Do I need a job queue, or can cron and a script carry this?","\nA script driven by cron is genuinely fine up to a few thousand jobs a day if the work is idempotent and a missed run is survivable. You need a real queue when you need per-job retry state, back-pressure, or partial progress across restarts — all three arrive together the first time a batch dies halfway. ",[18,4880,4882],{"href":4881},"\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002F","Running background jobs with Celery"," and the ",[18,4885,4887],{"href":4886},"\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002Fcelery-vs-rq-vs-arq\u002F","Celery versus RQ versus arq comparison"," cover the choice; arq is the natural fit for an async-native pipeline like this one.",[14,4890,4891,4894,4895,4897,4898,4902],{},[343,4892,4893],{},"How do I stop one customer's huge document from blowing the daily budget?","\nCount tokens before you send, not after. ",[187,4896,3585],{}," costs almost nothing and gives you the input size before you commit, so the queue can reject or downgrade an oversized job instead of discovering it on the invoice. Combine that with a per-customer daily ceiling in the same Redis ledger as the global cap, and the worst case becomes one rejected job with a clear error rather than an unbounded spend. ",[18,4899,4901],{"href":4900},"\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fcontrolling-llm-api-costs-in-production\u002F","Controlling LLM API costs in production"," covers the per-tenant accounting in detail.",[179,4904,4906],{"id":4905},"related","Related",[4908,4909,4910,4917,4923,4929,4935],"ul",{},[4911,4912,4913,4916],"li",{},[18,4914,4915],{"href":4900},"Controlling LLM API Costs in Production"," — per-tenant ledgers, token budgets, and the alerting that catches an overspend on day one rather than day thirty.",[4911,4918,4919,4922],{},[18,4920,4921],{"href":327},"Streaming LLM Responses Through FastAPI"," — the interactive half of this problem, for when a human is waiting on the tokens.",[4911,4924,4925,4928],{},[18,4926,4927],{"href":4881},"Running Background Jobs with Celery"," — the queue layer this pipeline sits on once cron stops being enough.",[4911,4930,4931,4934],{},[18,4932,4933],{"href":2410},"Retrying Failed HTTP Requests with tenacity"," — the general retry patterns behind Step 4, including decorator and context-manager forms.",[4911,4936,4937,4941],{},[18,4938,4940],{"href":4939},"\u002Fautomating-side-hustle-operations-with-apis\u002Fbuilding-zapier-alternatives-with-python\u002F","Building Zapier Alternatives with Python"," — where an AI step becomes one node in a larger automation graph you own end to end.",[4943,4944,4945],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}",{"title":209,"searchDepth":264,"depth":264,"links":4947},[4948,4949,4950,4951,4952,4953,4954,4955,4956,4957,4958,4959,4960],{"id":181,"depth":264,"text":182},{"id":334,"depth":264,"text":335},{"id":596,"depth":264,"text":597},{"id":1409,"depth":264,"text":1410},{"id":1980,"depth":264,"text":1981},{"id":2529,"depth":264,"text":2530},{"id":3042,"depth":264,"text":3043},{"id":4067,"depth":264,"text":4068},{"id":4249,"depth":264,"text":4250},{"id":4342,"depth":264,"text":4343},{"id":4713,"depth":264,"text":4714},{"id":4845,"depth":264,"text":4846},{"id":4905,"depth":264,"text":4906},"Wire LLM APIs into Python automation pipelines: async batching, schema-constrained extraction, retry and timeout handling, prompt versioning, and cost-aware queueing.","md",{"pageTitle":4964,"type":4965,"datePublished":4966,"dateModified":4966},"Automating AI Workflows with Python APIs in Production","guide","2026-07-23","\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis",{"title":5,"description":4961},"automating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Findex","X9FVeAwrrvko8Py5jynbT2yzpussSk8a_CzlkrxD6wQ",{"@context":4972,"@type":4973,"mainEntity":4974},"https:\u002F\u002Fschema.org","FAQPage",[4975,4980,4983,4986,4989],{"@type":4976,"name":4851,"acceptedAnswer":4977},"Question",{"@type":4978,"text":4979},"Answer","Using the worked example above — a shared 5,000-token prefix, a 1,000-token document, and 700 tokens out — you land at roughly $250 a month on a frontier model with prompt caching, $125 if the work can go through the batch endpoint, and about $50 if the jobs are mechanical enough for a small model. Compute is noise next to that: a single small container handles the whole volume. Budget 15% on top for retries and dead-lettered jobs, and instrument cost per successful job so you notice when that overhead grows.",{"@type":4976,"name":4857,"acceptedAnswer":4981},{"@type":4978,"text":4982},"Meter it, at least until your usage distribution stops surprising you. AI cost scales with input length, which scales with how much your heaviest customer pastes in, and a seat price averages that away in exactly the wrong direction — your worst-margin customer is also your happiest one. Start usage-based, cap the free tier by token volume rather than job count, and revisit once you have three months of data. The usage-based versus seat-based pricing trade-offs go deeper.",{"@type":4976,"name":4868,"acceptedAnswer":4984},{"@type":4978,"text":4985},"Lower than it feels, if you keep three seams clean: the client construction, the prompt registry, and the schema. Those are the only three places provider specifics appear in the code above; the queueing, retry, budget, and validation layers are provider-agnostic. What does not port is calibration — your prompt version, your max_tokens value, and your quality baseline all need re-measuring, which is a week of eval work rather than a rewrite. Keep an eval set of a few hundred labelled examples and the migration becomes a number rather than an argument.",{"@type":4976,"name":4877,"acceptedAnswer":4987},{"@type":4978,"text":4988},"A script driven by cron is genuinely fine up to a few thousand jobs a day if the work is idempotent and a missed run is survivable. You need a real queue when you need per-job retry state, back-pressure, or partial progress across restarts — all three arrive together the first time a batch dies halfway. Running background jobs with Celery and the Celery versus RQ versus arq comparison cover the choice; arq is the natural fit for an async-native pipeline like this one.",{"@type":4976,"name":4893,"acceptedAnswer":4990},{"@type":4978,"text":4991},"Count tokens before you send, not after. count_tokens costs almost nothing and gives you the input size before you commit, so the queue can reject or downgrade an oversized job instead of discovering it on the invoice. Combine that with a per-customer daily ceiling in the same Redis ledger as the global cap, and the worst case becomes one rejected job with a clear error rather than an unbounded spend. Controlling LLM API costs in production covers the per-tenant accounting in detail.",1784887027767]