[{"data":1,"prerenderedAt":2689},["ShallowReactive",2],{"page-\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fcontrolling-llm-api-costs-in-production\u002F":3,"faq-schema-\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fcontrolling-llm-api-costs-in-production\u002F":2668},{"id":4,"title":5,"body":6,"description":2658,"extension":2659,"meta":2660,"navigation":322,"path":2664,"seo":2665,"stem":2666,"__hash__":2667},"content\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fcontrolling-llm-api-costs-in-production\u002Findex.md","Controlling LLM API Costs in Production",{"type":7,"value":8,"toc":2645},"minimark",[9,13,23,26,31,34,37,152,156,159,167,282,983,991,995,998,1006,1354,1357,1431,1435,1438,1441,1542,1914,1917,1983,1987,1990,1998,2095,2477,2485,2489,2492,2500,2504,2507,2514,2518,2521,2546,2550,2558,2562,2569,2575,2581,2587,2604,2608,2641],[10,11,5],"h1",{"id":12},"controlling-llm-api-costs-in-production",[14,15,16,17,22],"p",{},"The decision this page resolves is where you put the ceiling. You can cap tokens per request, cap dollars per customer, cap which model a task may reach, or cap nothing and hope the invoice stays small. Most builders pick the last option by default, then discover on day forty that one trial account burned three months of margin pasting entire PDFs into a free endpoint. Part of the ",[18,19,21],"a",{"href":20},"\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002F","Automating AI Workflows with Python APIs"," guide, this page picks a specific stack of controls and ships the code for each.",[14,24,25],{},"The commercial frame matters more than the technical one. A model call is the only dependency whose unit cost is set by your customer's input rather than by your architecture. A call on a 40,000-token document costs twenty times a call on a 2,000-token one, and the customer paying a flat nineteen dollars a month has no idea they just triggered it. Every control below breaks that link.",[27,28,30],"h2",{"id":29},"the-four-levers-ranked-by-return","The four levers, ranked by return",[14,32,33],{},"There are only four things you can do about model spend: send fewer requests, send fewer tokens per request, pay less per token, or refuse to send past a limit. Deduplication and caching attack the first, token budgets the second, model tiering the third, and per-customer spend caps the fourth. Ranked by dollars saved per hour of engineering, that order is almost exactly right — deduplication is a hash lookup, and it routinely removes a quarter of production traffic in workflows where customers re-run the same job.",[14,35,36],{},"The figure below shows a realistic stack: a document-summarising background job with roughly 30% exact repeats, a stable 3,000-token system prompt, and a task mix where two thirds of jobs are classification rather than prose generation. Each step is a control you ship, not a modelling trick.",[38,39,47,48,47,52,47,56,47,63,47,71,47,78,47,88,47,93,47,96,47,102,47,106,47,109,47,115,47,119,47,122,47,128,47,132,47,135,47,140,47,144,47,147],"svg",{"viewBox":40,"role":41,"ariaLabelledBy":42,"xmlns":45,"style":46},"0 0 760 340","img",[43,44],"llmcost-funnel-t","llmcost-funnel-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;height:auto;margin:1.5rem 0;font-family:var(--font-sans);","\n  ",[49,50,51],"title",{"id":43},"Cumulative effect of four cost controls on model spend",[53,54,55],"desc",{"id":44},"A narrowing stack of five bars showing spend falling from an index of 100 to 92 after per-request token budgets, 64 after request deduplication, 50 after prompt caching, and 31 after model tiering.",[57,58],"rect",{"x":59,"y":59,"width":60,"height":61,"fill":62},"0","760","340","var(--c-surface)",[64,65,70],"text",{"x":66,"y":67,"fill":68,"style":69},"380","30","var(--c-text)","text-anchor:middle;font-size:14;font-family:var(--font-sans);","Each control multiplies against the one above it",[64,72,77],{"x":73,"y":74,"fill":75,"style":76},"740","48","var(--c-text-muted)","text-anchor:end;font-size:11;font-family:var(--font-sans);","spend index",[57,79],{"x":80,"y":81,"width":82,"height":83,"rx":84,"fill":85,"stroke":86,"style":87},"100","56","560","40","6","var(--c-surface-alt)","var(--c-coral)","stroke-width:2;",[64,89,92],{"x":66,"y":90,"fill":68,"style":91},"82","text-anchor:middle;font-size:13;font-family:var(--font-sans);","Raw model spend",[64,94,80],{"x":73,"y":90,"fill":68,"style":95},"text-anchor:end;font-size:13;font-family:var(--font-sans);",[57,97],{"x":98,"y":99,"width":100,"height":83,"rx":84,"fill":85,"stroke":101,"style":87},"122","108","515","var(--c-yellow)",[64,103,105],{"x":66,"y":104,"fill":68,"style":91},"134","Per-request token budget",[64,107,108],{"x":73,"y":104,"fill":68,"style":95},"92",[57,110],{"x":111,"y":112,"width":113,"height":83,"rx":84,"fill":85,"stroke":114,"style":87},"200","160","360","var(--c-teal)",[64,116,118],{"x":66,"y":117,"fill":68,"style":91},"186","Dedup on fingerprint",[64,120,121],{"x":73,"y":117,"fill":68,"style":95},"64",[57,123],{"x":124,"y":125,"width":126,"height":83,"rx":84,"fill":85,"stroke":127,"style":87},"240","212","280","var(--c-blue)",[64,129,131],{"x":66,"y":130,"fill":68,"style":91},"238","Prompt cache reads",[64,133,134],{"x":73,"y":130,"fill":68,"style":95},"50",[57,136],{"x":137,"y":138,"width":139,"height":83,"rx":84,"fill":85,"stroke":114,"style":87},"292","264","175",[64,141,143],{"x":66,"y":142,"fill":68,"style":91},"290","Model tiering",[64,145,146],{"x":73,"y":142,"fill":68,"style":95},"31",[64,148,151],{"x":66,"y":149,"fill":75,"style":150},"326","text-anchor:middle;font-size:12;font-family:var(--font-sans);","Same workload, same output quality — roughly 31 cents on the original dollar",[27,153,155],{"id":154},"budget-tokens-before-you-send-not-after","Budget tokens before you send, not after",[14,157,158],{},"The mistake almost everyone makes is measuring cost from the response. By then you have already paid. Token budgeting means deciding, before the request leaves your process, whether this call is allowed to happen at the size it wants. The Anthropic SDK exposes a token counter for exactly this, and it is free to call.",[14,160,161,162,166],{},"Two numbers govern each call. ",[163,164,165],"code",{},"max_tokens"," caps the output side and is enforced by the API, so it is your hard ceiling on the expensive half of the bill — output tokens cost five times input tokens on every current model. The input side has no server-side cap, so you count first and either trim the context or reject the job. Rejecting is usually correct in a background worker: a job needing 40,000 tokens of context belongs to a customer who should be on a different plan.",[38,168,47,173,47,176,47,179,47,183,47,201,47,205,47,209,47,214,47,217,47,221,47,224,47,228,47,233,47,235,47,237,47,243,47,247,47,251,47,255,47,259,47,264,47,269,47,272,47,275,47,278],{"viewBox":169,"role":41,"ariaLabelledBy":170,"xmlns":45,"style":46},"0 0 720 330",[171,172],"llmcost-gate-t","llmcost-gate-d",[49,174,175],{"id":171},"Sequence of a token-budgeted model call",[53,177,178],{"id":172},"A worker calls the token counter, receives an input token count, trims or rejects if over budget, then calls the model API with a capped max_tokens and writes the reported usage to a cost ledger.",[57,180],{"x":59,"y":59,"width":181,"height":182,"fill":62},"720","330",[184,185,186,187,47],"defs",{},"\n    ",[188,189,196,197,186],"marker",{"id":190,"viewBox":191,"refX":192,"refY":193,"markerWidth":194,"markerHeight":194,"orient":195},"llmcost-gate-arrow","0 0 10 10","9","5","7","auto-start-reverse","\n      ",[198,199],"path",{"d":200,"fill":75},"M0 0 L10 5 L0 10 z",[64,202,204],{"x":113,"y":203,"fill":68,"style":69},"28","Count first, spend second",[57,206],{"x":134,"y":74,"width":207,"height":208,"rx":84,"fill":85,"stroke":127,"style":87},"140","34",[64,210,213],{"x":211,"y":212,"fill":68,"style":150},"120","70","Worker",[57,215],{"x":216,"y":74,"width":207,"height":208,"rx":84,"fill":85,"stroke":114,"style":87},"300",[64,218,220],{"x":219,"y":212,"fill":68,"style":150},"370","Token counter",[57,222],{"x":223,"y":74,"width":207,"height":208,"rx":84,"fill":85,"stroke":86,"style":87},"540",[64,225,227],{"x":226,"y":212,"fill":68,"style":150},"610","Model API",[229,230],"line",{"x1":211,"y1":231,"x2":211,"y2":137,"stroke":232,"style":87},"88","var(--c-border)",[229,234],{"x1":219,"y1":231,"x2":219,"y2":137,"stroke":232,"style":87},[229,236],{"x1":226,"y1":231,"x2":226,"y2":137,"stroke":232,"style":87},[64,238,242],{"x":239,"y":240,"fill":75,"style":241},"245","112","text-anchor:middle;font-size:11;font-family:var(--font-sans);","count_tokens(prompt)",[229,244],{"x1":211,"y1":211,"x2":245,"y2":211,"stroke":75,"style":246},"366","stroke-width:2;marker-end:url(#llmcost-gate-arrow);",[64,248,250],{"x":239,"y":249,"fill":75,"style":241},"146","input_tokens = 14,200",[229,252],{"x1":219,"y1":253,"x2":254,"y2":253,"stroke":75,"style":246},"154","124",[57,256],{"x":67,"y":257,"width":258,"height":83,"rx":84,"fill":85,"stroke":101,"style":87},"176","250",[64,260,263],{"x":261,"y":262,"fill":68,"style":150},"155","201","Over 12,000: trim or reject",[64,265,268],{"x":266,"y":267,"fill":75,"style":241},"365","230","messages.create(max_tokens=1024)",[229,270],{"x1":211,"y1":130,"x2":271,"y2":130,"stroke":75,"style":246},"606",[64,273,274],{"x":266,"y":138,"fill":75,"style":241},"usage.input_tokens to ledger",[229,276],{"x1":226,"y1":277,"x2":254,"y2":277,"stroke":75,"style":246},"272",[64,279,281],{"x":113,"y":280,"fill":75,"style":150},"316","The counter call is free; the call you avoid is the one you would have paid for",[283,284,289],"pre",{"className":285,"code":286,"language":287,"meta":288,"style":288},"language-python shiki shiki-themes github-light github-dark","import os\nfrom dataclasses import dataclass\n\nimport anthropic\n\nclient = anthropic.AsyncAnthropic()\n\nMODEL = os.getenv(\"LLM_MODEL\", \"claude-haiku-4-5\")\nMAX_INPUT_TOKENS = int(os.getenv(\"LLM_MAX_INPUT_TOKENS\", \"12000\"))\nMAX_OUTPUT_TOKENS = int(os.getenv(\"LLM_MAX_OUTPUT_TOKENS\", \"1024\"))\nINPUT_PER_MTOK = float(os.getenv(\"LLM_INPUT_PER_MTOK\", \"1.00\"))\nOUTPUT_PER_MTOK = float(os.getenv(\"LLM_OUTPUT_PER_MTOK\", \"5.00\"))\n\n\nclass BudgetExceeded(RuntimeError):\n    \"\"\"Raised before any billable request is sent.\"\"\"\n\n\n@dataclass(frozen=True, slots=True)\nclass CallResult:\n    text: str\n    input_tokens: int\n    output_tokens: int\n\n    @property\n    def usd(self) -> float:\n        return (\n            self.input_tokens * INPUT_PER_MTOK\n            + self.output_tokens * OUTPUT_PER_MTOK\n        ) \u002F 1_000_000\n\n\nasync def budgeted_call(system: str, user_text: str) -> CallResult:\n    messages = [{\"role\": \"user\", \"content\": user_text}]\n    counted = await client.messages.count_tokens(\n        model=MODEL, system=system, messages=messages\n    )\n    if counted.input_tokens > MAX_INPUT_TOKENS:\n        raise BudgetExceeded(\n            f\"{counted.input_tokens} input tokens exceeds {MAX_INPUT_TOKENS}\"\n        )\n\n    message = await client.messages.create(\n        model=MODEL,\n        system=system,\n        messages=messages,\n        max_tokens=MAX_OUTPUT_TOKENS,\n    )\n    return CallResult(\n        text=\"\".join(b.text for b in message.content if b.type == \"text\"),\n        input_tokens=message.usage.input_tokens,\n        output_tokens=message.usage.output_tokens,\n    )\n","python","",[163,290,291,303,317,324,332,337,349,354,380,405,427,450,472,477,482,501,507,512,517,545,556,565,574,582,587,596,613,622,637,654,666,671,676,702,730,744,772,778,795,804,831,837,842,855,867,878,889,901,906,915,956,967,978],{"__ignoreMap":288},[292,293,295,299],"span",{"class":229,"line":294},1,[292,296,298],{"class":297},"szBVR","import",[292,300,302],{"class":301},"sVt8B"," os\n",[292,304,306,309,312,314],{"class":229,"line":305},2,[292,307,308],{"class":297},"from",[292,310,311],{"class":301}," dataclasses ",[292,313,298],{"class":297},[292,315,316],{"class":301}," dataclass\n",[292,318,320],{"class":229,"line":319},3,[292,321,323],{"emptyLinePlaceholder":322},true,"\n",[292,325,327,329],{"class":229,"line":326},4,[292,328,298],{"class":297},[292,330,331],{"class":301}," anthropic\n",[292,333,335],{"class":229,"line":334},5,[292,336,323],{"emptyLinePlaceholder":322},[292,338,340,343,346],{"class":229,"line":339},6,[292,341,342],{"class":301},"client ",[292,344,345],{"class":297},"=",[292,347,348],{"class":301}," anthropic.AsyncAnthropic()\n",[292,350,352],{"class":229,"line":351},7,[292,353,323],{"emptyLinePlaceholder":322},[292,355,357,361,364,367,371,374,377],{"class":229,"line":356},8,[292,358,360],{"class":359},"sj4cs","MODEL",[292,362,363],{"class":297}," =",[292,365,366],{"class":301}," os.getenv(",[292,368,370],{"class":369},"sZZnC","\"LLM_MODEL\"",[292,372,373],{"class":301},", ",[292,375,376],{"class":369},"\"claude-haiku-4-5\"",[292,378,379],{"class":301},")\n",[292,381,383,386,388,391,394,397,399,402],{"class":229,"line":382},9,[292,384,385],{"class":359},"MAX_INPUT_TOKENS",[292,387,363],{"class":297},[292,389,390],{"class":359}," int",[292,392,393],{"class":301},"(os.getenv(",[292,395,396],{"class":369},"\"LLM_MAX_INPUT_TOKENS\"",[292,398,373],{"class":301},[292,400,401],{"class":369},"\"12000\"",[292,403,404],{"class":301},"))\n",[292,406,408,411,413,415,417,420,422,425],{"class":229,"line":407},10,[292,409,410],{"class":359},"MAX_OUTPUT_TOKENS",[292,412,363],{"class":297},[292,414,390],{"class":359},[292,416,393],{"class":301},[292,418,419],{"class":369},"\"LLM_MAX_OUTPUT_TOKENS\"",[292,421,373],{"class":301},[292,423,424],{"class":369},"\"1024\"",[292,426,404],{"class":301},[292,428,430,433,435,438,440,443,445,448],{"class":229,"line":429},11,[292,431,432],{"class":359},"INPUT_PER_MTOK",[292,434,363],{"class":297},[292,436,437],{"class":359}," float",[292,439,393],{"class":301},[292,441,442],{"class":369},"\"LLM_INPUT_PER_MTOK\"",[292,444,373],{"class":301},[292,446,447],{"class":369},"\"1.00\"",[292,449,404],{"class":301},[292,451,453,456,458,460,462,465,467,470],{"class":229,"line":452},12,[292,454,455],{"class":359},"OUTPUT_PER_MTOK",[292,457,363],{"class":297},[292,459,437],{"class":359},[292,461,393],{"class":301},[292,463,464],{"class":369},"\"LLM_OUTPUT_PER_MTOK\"",[292,466,373],{"class":301},[292,468,469],{"class":369},"\"5.00\"",[292,471,404],{"class":301},[292,473,475],{"class":229,"line":474},13,[292,476,323],{"emptyLinePlaceholder":322},[292,478,480],{"class":229,"line":479},14,[292,481,323],{"emptyLinePlaceholder":322},[292,483,485,488,492,495,498],{"class":229,"line":484},15,[292,486,487],{"class":297},"class",[292,489,491],{"class":490},"sScJk"," BudgetExceeded",[292,493,494],{"class":301},"(",[292,496,497],{"class":359},"RuntimeError",[292,499,500],{"class":301},"):\n",[292,502,504],{"class":229,"line":503},16,[292,505,506],{"class":369},"    \"\"\"Raised before any billable request is sent.\"\"\"\n",[292,508,510],{"class":229,"line":509},17,[292,511,323],{"emptyLinePlaceholder":322},[292,513,515],{"class":229,"line":514},18,[292,516,323],{"emptyLinePlaceholder":322},[292,518,520,523,525,529,531,534,536,539,541,543],{"class":229,"line":519},19,[292,521,522],{"class":490},"@dataclass",[292,524,494],{"class":301},[292,526,528],{"class":527},"s4XuR","frozen",[292,530,345],{"class":297},[292,532,533],{"class":359},"True",[292,535,373],{"class":301},[292,537,538],{"class":527},"slots",[292,540,345],{"class":297},[292,542,533],{"class":359},[292,544,379],{"class":301},[292,546,548,550,553],{"class":229,"line":547},20,[292,549,487],{"class":297},[292,551,552],{"class":490}," CallResult",[292,554,555],{"class":301},":\n",[292,557,559,562],{"class":229,"line":558},21,[292,560,561],{"class":301},"    text: ",[292,563,564],{"class":359},"str\n",[292,566,568,571],{"class":229,"line":567},22,[292,569,570],{"class":301},"    input_tokens: ",[292,572,573],{"class":359},"int\n",[292,575,577,580],{"class":229,"line":576},23,[292,578,579],{"class":301},"    output_tokens: ",[292,581,573],{"class":359},[292,583,585],{"class":229,"line":584},24,[292,586,323],{"emptyLinePlaceholder":322},[292,588,590,593],{"class":229,"line":589},25,[292,591,592],{"class":490},"    @",[292,594,595],{"class":359},"property\n",[292,597,599,602,605,608,611],{"class":229,"line":598},26,[292,600,601],{"class":297},"    def",[292,603,604],{"class":490}," usd",[292,606,607],{"class":301},"(self) -> ",[292,609,610],{"class":359},"float",[292,612,555],{"class":301},[292,614,616,619],{"class":229,"line":615},27,[292,617,618],{"class":297},"        return",[292,620,621],{"class":301}," (\n",[292,623,625,628,631,634],{"class":229,"line":624},28,[292,626,627],{"class":359},"            self",[292,629,630],{"class":301},".input_tokens ",[292,632,633],{"class":297},"*",[292,635,636],{"class":359}," INPUT_PER_MTOK\n",[292,638,640,643,646,649,651],{"class":229,"line":639},29,[292,641,642],{"class":297},"            +",[292,644,645],{"class":359}," self",[292,647,648],{"class":301},".output_tokens ",[292,650,633],{"class":297},[292,652,653],{"class":359}," OUTPUT_PER_MTOK\n",[292,655,657,660,663],{"class":229,"line":656},30,[292,658,659],{"class":301},"        ) ",[292,661,662],{"class":297},"\u002F",[292,664,665],{"class":359}," 1_000_000\n",[292,667,669],{"class":229,"line":668},31,[292,670,323],{"emptyLinePlaceholder":322},[292,672,674],{"class":229,"line":673},32,[292,675,323],{"emptyLinePlaceholder":322},[292,677,679,682,685,688,691,694,697,699],{"class":229,"line":678},33,[292,680,681],{"class":297},"async",[292,683,684],{"class":297}," def",[292,686,687],{"class":490}," budgeted_call",[292,689,690],{"class":301},"(system: ",[292,692,693],{"class":359},"str",[292,695,696],{"class":301},", user_text: ",[292,698,693],{"class":359},[292,700,701],{"class":301},") -> CallResult:\n",[292,703,705,708,710,713,716,719,722,724,727],{"class":229,"line":704},34,[292,706,707],{"class":301},"    messages ",[292,709,345],{"class":297},[292,711,712],{"class":301}," [{",[292,714,715],{"class":369},"\"role\"",[292,717,718],{"class":301},": ",[292,720,721],{"class":369},"\"user\"",[292,723,373],{"class":301},[292,725,726],{"class":369},"\"content\"",[292,728,729],{"class":301},": user_text}]\n",[292,731,733,736,738,741],{"class":229,"line":732},35,[292,734,735],{"class":301},"    counted ",[292,737,345],{"class":297},[292,739,740],{"class":297}," await",[292,742,743],{"class":301}," client.messages.count_tokens(\n",[292,745,747,750,752,754,756,759,761,764,767,769],{"class":229,"line":746},36,[292,748,749],{"class":527},"        model",[292,751,345],{"class":297},[292,753,360],{"class":359},[292,755,373],{"class":301},[292,757,758],{"class":527},"system",[292,760,345],{"class":297},[292,762,763],{"class":301},"system, ",[292,765,766],{"class":527},"messages",[292,768,345],{"class":297},[292,770,771],{"class":301},"messages\n",[292,773,775],{"class":229,"line":774},37,[292,776,777],{"class":301},"    )\n",[292,779,781,784,787,790,793],{"class":229,"line":780},38,[292,782,783],{"class":297},"    if",[292,785,786],{"class":301}," counted.input_tokens ",[292,788,789],{"class":297},">",[292,791,792],{"class":359}," MAX_INPUT_TOKENS",[292,794,555],{"class":301},[292,796,798,801],{"class":229,"line":797},39,[292,799,800],{"class":297},"        raise",[292,802,803],{"class":301}," BudgetExceeded(\n",[292,805,807,810,813,816,819,822,825,828],{"class":229,"line":806},40,[292,808,809],{"class":297},"            f",[292,811,812],{"class":369},"\"",[292,814,815],{"class":359},"{",[292,817,818],{"class":301},"counted.input_tokens",[292,820,821],{"class":359},"}",[292,823,824],{"class":369}," input tokens exceeds ",[292,826,827],{"class":359},"{MAX_INPUT_TOKENS}",[292,829,830],{"class":369},"\"\n",[292,832,834],{"class":229,"line":833},41,[292,835,836],{"class":301},"        )\n",[292,838,840],{"class":229,"line":839},42,[292,841,323],{"emptyLinePlaceholder":322},[292,843,845,848,850,852],{"class":229,"line":844},43,[292,846,847],{"class":301},"    message ",[292,849,345],{"class":297},[292,851,740],{"class":297},[292,853,854],{"class":301}," client.messages.create(\n",[292,856,858,860,862,864],{"class":229,"line":857},44,[292,859,749],{"class":527},[292,861,345],{"class":297},[292,863,360],{"class":359},[292,865,866],{"class":301},",\n",[292,868,870,873,875],{"class":229,"line":869},45,[292,871,872],{"class":527},"        system",[292,874,345],{"class":297},[292,876,877],{"class":301},"system,\n",[292,879,881,884,886],{"class":229,"line":880},46,[292,882,883],{"class":527},"        messages",[292,885,345],{"class":297},[292,887,888],{"class":301},"messages,\n",[292,890,892,895,897,899],{"class":229,"line":891},47,[292,893,894],{"class":527},"        max_tokens",[292,896,345],{"class":297},[292,898,410],{"class":359},[292,900,866],{"class":301},[292,902,904],{"class":229,"line":903},48,[292,905,777],{"class":301},[292,907,909,912],{"class":229,"line":908},49,[292,910,911],{"class":297},"    return",[292,913,914],{"class":301}," CallResult(\n",[292,916,918,921,923,926,929,932,935,938,941,944,947,950,953],{"class":229,"line":917},50,[292,919,920],{"class":527},"        text",[292,922,345],{"class":297},[292,924,925],{"class":369},"\"\"",[292,927,928],{"class":301},".join(b.text ",[292,930,931],{"class":297},"for",[292,933,934],{"class":301}," b ",[292,936,937],{"class":297},"in",[292,939,940],{"class":301}," message.content ",[292,942,943],{"class":297},"if",[292,945,946],{"class":301}," b.type ",[292,948,949],{"class":297},"==",[292,951,952],{"class":369}," \"text\"",[292,954,955],{"class":301},"),\n",[292,957,959,962,964],{"class":229,"line":958},51,[292,960,961],{"class":527},"        input_tokens",[292,963,345],{"class":297},[292,965,966],{"class":301},"message.usage.input_tokens,\n",[292,968,970,973,975],{"class":229,"line":969},52,[292,971,972],{"class":527},"        output_tokens",[292,974,345],{"class":297},[292,976,977],{"class":301},"message.usage.output_tokens,\n",[292,979,981],{"class":229,"line":980},53,[292,982,777],{"class":301},[14,984,985,986,990],{},"Read the price constants from the environment rather than hardcoding them: vendor pricing changes, and a ledger that silently reports last year's rates is worse than no ledger. The reasoning behind ",[18,987,989],{"href":988},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fcalculating-cost-per-api-request\u002F","calculating cost per API request"," for your own endpoints applies doubly here, because the input size is not yours to control.",[27,992,994],{"id":993},"caching-and-deduplication-two-different-wins","Caching and deduplication: two different wins",[14,996,997],{},"These get conflated constantly. Deduplication means you never call the model, because you already hold the answer for this exact input. Prompt caching means you do call it, but the shared prefix bills at roughly a tenth of the normal input rate. Ship both — they compose.",[14,999,1000,1001,1005],{},"Deduplication needs a fingerprint covering everything that changes the answer: model ID, prompt version, system text, and the normalised user input. Leave the prompt version out and your first prompt improvement silently serves stale results to every returning customer. Store the result in Redis with a TTL tied to how long the answer stays true; the trade-offs match those in ",[18,1002,1004],{"href":1003},"\u002Fscaling-and-operating-production-python-apis\u002Fcaching-python-api-responses-with-redis\u002Fcache-invalidation-strategies\u002F","cache invalidation strategies",".",[283,1007,1009],{"className":285,"code":1008,"language":287,"meta":288,"style":288},"import hashlib\nimport json\nimport os\n\nimport redis.asyncio as redis\n\nr = redis.from_url(os.getenv(\"REDIS_URL\", \"redis:\u002F\u002Flocalhost:6379\u002F0\"))\nPROMPT_VERSION = os.getenv(\"LLM_PROMPT_VERSION\", \"v3\")\nDEDUP_TTL = int(os.getenv(\"LLM_DEDUP_TTL_SECONDS\", \"86400\"))\n\n\ndef fingerprint(system: str, user_text: str) -> str:\n    digest = hashlib.sha256()\n    for part in (MODEL, PROMPT_VERSION, system, user_text.strip()):\n        digest.update(part.encode(\"utf-8\"))\n        digest.update(b\"\\x00\")\n    return f\"llm:{digest.hexdigest()}\"\n\n\nasync def cached_call(system: str, user_text: str) -> CallResult:\n    key = fingerprint(system, user_text)\n    if hit := await r.get(key):\n        payload = json.loads(hit)\n        return CallResult(payload[\"text\"], 0, 0)  # zero cost, zero tokens\n\n    result = await budgeted_call(system, user_text)\n    await r.set(key, json.dumps({\"text\": result.text}), ex=DEDUP_TTL)\n    return result\n",[163,1010,1011,1018,1025,1031,1035,1048,1052,1072,1091,1112,1116,1120,1143,1153,1175,1185,1202,1221,1225,1229,1248,1258,1273,1283,1309,1313,1325,1347],{"__ignoreMap":288},[292,1012,1013,1015],{"class":229,"line":294},[292,1014,298],{"class":297},[292,1016,1017],{"class":301}," hashlib\n",[292,1019,1020,1022],{"class":229,"line":305},[292,1021,298],{"class":297},[292,1023,1024],{"class":301}," json\n",[292,1026,1027,1029],{"class":229,"line":319},[292,1028,298],{"class":297},[292,1030,302],{"class":301},[292,1032,1033],{"class":229,"line":326},[292,1034,323],{"emptyLinePlaceholder":322},[292,1036,1037,1039,1042,1045],{"class":229,"line":334},[292,1038,298],{"class":297},[292,1040,1041],{"class":301}," redis.asyncio ",[292,1043,1044],{"class":297},"as",[292,1046,1047],{"class":301}," redis\n",[292,1049,1050],{"class":229,"line":339},[292,1051,323],{"emptyLinePlaceholder":322},[292,1053,1054,1057,1059,1062,1065,1067,1070],{"class":229,"line":351},[292,1055,1056],{"class":301},"r ",[292,1058,345],{"class":297},[292,1060,1061],{"class":301}," redis.from_url(os.getenv(",[292,1063,1064],{"class":369},"\"REDIS_URL\"",[292,1066,373],{"class":301},[292,1068,1069],{"class":369},"\"redis:\u002F\u002Flocalhost:6379\u002F0\"",[292,1071,404],{"class":301},[292,1073,1074,1077,1079,1081,1084,1086,1089],{"class":229,"line":356},[292,1075,1076],{"class":359},"PROMPT_VERSION",[292,1078,363],{"class":297},[292,1080,366],{"class":301},[292,1082,1083],{"class":369},"\"LLM_PROMPT_VERSION\"",[292,1085,373],{"class":301},[292,1087,1088],{"class":369},"\"v3\"",[292,1090,379],{"class":301},[292,1092,1093,1096,1098,1100,1102,1105,1107,1110],{"class":229,"line":382},[292,1094,1095],{"class":359},"DEDUP_TTL",[292,1097,363],{"class":297},[292,1099,390],{"class":359},[292,1101,393],{"class":301},[292,1103,1104],{"class":369},"\"LLM_DEDUP_TTL_SECONDS\"",[292,1106,373],{"class":301},[292,1108,1109],{"class":369},"\"86400\"",[292,1111,404],{"class":301},[292,1113,1114],{"class":229,"line":407},[292,1115,323],{"emptyLinePlaceholder":322},[292,1117,1118],{"class":229,"line":429},[292,1119,323],{"emptyLinePlaceholder":322},[292,1121,1122,1125,1128,1130,1132,1134,1136,1139,1141],{"class":229,"line":452},[292,1123,1124],{"class":297},"def",[292,1126,1127],{"class":490}," fingerprint",[292,1129,690],{"class":301},[292,1131,693],{"class":359},[292,1133,696],{"class":301},[292,1135,693],{"class":359},[292,1137,1138],{"class":301},") -> ",[292,1140,693],{"class":359},[292,1142,555],{"class":301},[292,1144,1145,1148,1150],{"class":229,"line":474},[292,1146,1147],{"class":301},"    digest ",[292,1149,345],{"class":297},[292,1151,1152],{"class":301}," hashlib.sha256()\n",[292,1154,1155,1158,1161,1163,1166,1168,1170,1172],{"class":229,"line":479},[292,1156,1157],{"class":297},"    for",[292,1159,1160],{"class":301}," part ",[292,1162,937],{"class":297},[292,1164,1165],{"class":301}," (",[292,1167,360],{"class":359},[292,1169,373],{"class":301},[292,1171,1076],{"class":359},[292,1173,1174],{"class":301},", system, user_text.strip()):\n",[292,1176,1177,1180,1183],{"class":229,"line":484},[292,1178,1179],{"class":301},"        digest.update(part.encode(",[292,1181,1182],{"class":369},"\"utf-8\"",[292,1184,404],{"class":301},[292,1186,1187,1190,1193,1195,1198,1200],{"class":229,"line":503},[292,1188,1189],{"class":301},"        digest.update(",[292,1191,1192],{"class":297},"b",[292,1194,812],{"class":369},[292,1196,1197],{"class":359},"\\x00",[292,1199,812],{"class":369},[292,1201,379],{"class":301},[292,1203,1204,1206,1209,1212,1214,1217,1219],{"class":229,"line":509},[292,1205,911],{"class":297},[292,1207,1208],{"class":297}," f",[292,1210,1211],{"class":369},"\"llm:",[292,1213,815],{"class":359},[292,1215,1216],{"class":301},"digest.hexdigest()",[292,1218,821],{"class":359},[292,1220,830],{"class":369},[292,1222,1223],{"class":229,"line":514},[292,1224,323],{"emptyLinePlaceholder":322},[292,1226,1227],{"class":229,"line":519},[292,1228,323],{"emptyLinePlaceholder":322},[292,1230,1231,1233,1235,1238,1240,1242,1244,1246],{"class":229,"line":547},[292,1232,681],{"class":297},[292,1234,684],{"class":297},[292,1236,1237],{"class":490}," cached_call",[292,1239,690],{"class":301},[292,1241,693],{"class":359},[292,1243,696],{"class":301},[292,1245,693],{"class":359},[292,1247,701],{"class":301},[292,1249,1250,1253,1255],{"class":229,"line":558},[292,1251,1252],{"class":301},"    key ",[292,1254,345],{"class":297},[292,1256,1257],{"class":301}," fingerprint(system, user_text)\n",[292,1259,1260,1262,1265,1268,1270],{"class":229,"line":567},[292,1261,783],{"class":297},[292,1263,1264],{"class":301}," hit ",[292,1266,1267],{"class":297},":=",[292,1269,740],{"class":297},[292,1271,1272],{"class":301}," r.get(key):\n",[292,1274,1275,1278,1280],{"class":229,"line":576},[292,1276,1277],{"class":301},"        payload ",[292,1279,345],{"class":297},[292,1281,1282],{"class":301}," json.loads(hit)\n",[292,1284,1285,1287,1290,1293,1296,1298,1300,1302,1305],{"class":229,"line":584},[292,1286,618],{"class":297},[292,1288,1289],{"class":301}," CallResult(payload[",[292,1291,1292],{"class":369},"\"text\"",[292,1294,1295],{"class":301},"], ",[292,1297,59],{"class":359},[292,1299,373],{"class":301},[292,1301,59],{"class":359},[292,1303,1304],{"class":301},")  ",[292,1306,1308],{"class":1307},"sJ8bj","# zero cost, zero tokens\n",[292,1310,1311],{"class":229,"line":589},[292,1312,323],{"emptyLinePlaceholder":322},[292,1314,1315,1318,1320,1322],{"class":229,"line":598},[292,1316,1317],{"class":301},"    result ",[292,1319,345],{"class":297},[292,1321,740],{"class":297},[292,1323,1324],{"class":301}," budgeted_call(system, user_text)\n",[292,1326,1327,1330,1333,1335,1338,1341,1343,1345],{"class":229,"line":615},[292,1328,1329],{"class":297},"    await",[292,1331,1332],{"class":301}," r.set(key, json.dumps({",[292,1334,1292],{"class":369},[292,1336,1337],{"class":301},": result.text}), ",[292,1339,1340],{"class":527},"ex",[292,1342,345],{"class":297},[292,1344,1095],{"class":359},[292,1346,379],{"class":301},[292,1348,1349,1351],{"class":229,"line":624},[292,1350,911],{"class":297},[292,1352,1353],{"class":301}," result\n",[14,1355,1356],{},"Prompt caching is a separate switch on the request itself. Mark the stable prefix — system prompt, few-shot examples, the schema you always send — with a cache breakpoint, and keep every volatile byte after it. A timestamp interpolated into the system prompt invalidates the whole prefix on every call, which is the usual reason builders report that caching \"does nothing\".",[38,1358,47,1363,47,1366,47,1369,47,1373,47,1377,47,1380,47,1386,47,1390,47,1393,47,1396,47,1400,47,1405,47,1408,47,1411,47,1416,47,1421,47,1424,47,1427],{"viewBox":1359,"role":41,"ariaLabelledBy":1360,"xmlns":45,"style":46},"0 0 700 320",[1361,1362],"llmcost-cache-t","llmcost-cache-d",[49,1364,1365],{"id":1361},"Cost per million input tokens under three caching outcomes",[53,1367,1368],{"id":1362},"Bar chart on a five-dollar-per-million base rate: uncached input costs five dollars, a cache write costs six dollars twenty-five, and a cache read costs fifty cents.",[57,1370],{"x":59,"y":59,"width":1371,"height":1372,"fill":62},"700","320",[64,1374,1376],{"x":1375,"y":67,"fill":68,"style":69},"350","Dollars per 1M input tokens at a $5.00 base rate",[229,1378],{"x1":212,"y1":258,"x2":1379,"y2":258,"stroke":232,"style":87},"640",[57,1381],{"x":1382,"y":240,"width":1383,"height":1384,"rx":1385,"fill":85,"stroke":86,"style":87},"110","90","138","4",[64,1387,1389],{"x":261,"y":1388,"fill":68,"style":91},"104","$5.00",[64,1391,1392],{"x":261,"y":277,"fill":68,"style":150},"Uncached",[64,1394,1395],{"x":261,"y":142,"fill":75,"style":241},"every call",[57,1397],{"x":216,"y":1398,"width":1383,"height":1399,"rx":1385,"fill":85,"stroke":101,"style":87},"77","173",[64,1401,1404],{"x":1402,"y":1403,"fill":68,"style":91},"345","69","$6.25",[64,1406,1407],{"x":1402,"y":277,"fill":68,"style":150},"Cache write",[64,1409,1410],{"x":1402,"y":142,"fill":75,"style":241},"first call only",[57,1412],{"x":1413,"y":1414,"width":1383,"height":1415,"rx":1385,"fill":85,"stroke":114,"style":87},"490","236","14",[64,1417,1420],{"x":1418,"y":1419,"fill":68,"style":91},"535","228","$0.50",[64,1422,1423],{"x":1418,"y":277,"fill":68,"style":150},"Cache read",[64,1425,1426],{"x":1418,"y":142,"fill":75,"style":241},"every call after",[64,1428,1430],{"x":1375,"y":1429,"fill":75,"style":150},"312","Two calls on the same prefix already beat two uncached calls",[27,1432,1434],{"id":1433},"model-tiering-route-down-never-up","Model tiering: route down, never up",[14,1436,1437],{},"Price spreads across the tiers are wide enough that routing is the largest structural saving available. On a representative job of 2,000 input tokens and 400 output tokens, the small model costs $0.004, the mid tier $0.012, and the frontier tier $0.020. At 50,000 jobs a month that is $200 against $1,000 — the difference between a healthy gross margin and an apology to your accountant.",[14,1439,1440],{},"Route down by default and promote a task only when a real evaluation shows the cheap tier failing. Classification, extraction, tagging, and short structured summaries almost never need a frontier model; multi-step reasoning, long tool chains, and anything a customer reads verbatim usually do. Encode that as data, not scattered conditionals.",[38,1442,47,1447,47,1450,47,1453,47,1455,47,1459,47,1463,47,1467,47,1471,47,1473,47,1476,47,1479,47,1483,47,1487,47,1489,47,1492,47,1495,47,1498,47,1501,47,1504,47,1508,47,1510,47,1512,47,1515,47,1519,47,1523,47,1525,47,1528,47,1531,47,1533,47,1536,47,1539],{"viewBox":1443,"role":41,"ariaLabelledBy":1444,"xmlns":45,"style":46},"0 0 760 330",[1445,1446],"llmcost-tier-t","llmcost-tier-d",[49,1448,1449],{"id":1445},"Decision tree routing a job to a model tier",[53,1451,1452],{"id":1446},"An incoming job branches on three conditions — fixed schema output, customer-facing prose, or multi-step reasoning — routing respectively to a small, mid, and frontier model with their per-million token prices.",[57,1454],{"x":59,"y":59,"width":60,"height":182,"fill":62},[64,1456,1458],{"x":66,"y":1457,"fill":68,"style":69},"24","Pick the cheapest tier that passes your eval",[57,1460],{"x":1461,"y":83,"width":207,"height":83,"rx":1462,"fill":85,"stroke":127,"style":87},"310","8",[64,1464,1466],{"x":66,"y":1465,"fill":68,"style":91},"65","Incoming job",[229,1468],{"x1":66,"y1":1469,"x2":1382,"y2":1470,"stroke":232,"style":87},"80","130",[229,1472],{"x1":66,"y1":1469,"x2":66,"y2":1470,"stroke":232,"style":87},[229,1474],{"x1":66,"y1":1469,"x2":1475,"y2":1470,"stroke":232,"style":87},"650",[57,1477],{"x":67,"y":1470,"width":112,"height":1478,"rx":1462,"fill":62,"stroke":114,"style":87},"54",[64,1480,1482],{"x":1382,"y":1481,"fill":68,"style":150},"152","Fixed schema out?",[64,1484,1486],{"x":1382,"y":1485,"fill":75,"style":241},"170","classify \u002F extract",[57,1488],{"x":216,"y":1470,"width":112,"height":1478,"rx":1462,"fill":62,"stroke":101,"style":87},[64,1490,1491],{"x":66,"y":1481,"fill":68,"style":150},"Customer reads it?",[64,1493,1494],{"x":66,"y":1485,"fill":75,"style":241},"drafts \u002F summaries",[57,1496],{"x":1497,"y":1470,"width":112,"height":1478,"rx":1462,"fill":62,"stroke":86,"style":87},"570",[64,1499,1500],{"x":1475,"y":1481,"fill":68,"style":150},"Multi-step work?",[64,1502,1503],{"x":1475,"y":1485,"fill":75,"style":241},"tool chains",[229,1505],{"x1":1382,"y1":1506,"x2":1382,"y2":1507,"stroke":232,"style":87},"184","232",[229,1509],{"x1":66,"y1":1506,"x2":66,"y2":1507,"stroke":232,"style":87},[229,1511],{"x1":1475,"y1":1506,"x2":1475,"y2":1507,"stroke":232,"style":87},[57,1513],{"x":67,"y":1507,"width":112,"height":1514,"rx":1462,"fill":85,"stroke":114,"style":87},"58",[64,1516,1518],{"x":1382,"y":1517,"fill":68,"style":91},"256","Small tier",[64,1520,1522],{"x":1382,"y":1521,"fill":75,"style":241},"276","$1 in \u002F $5 out",[57,1524],{"x":216,"y":1507,"width":112,"height":1514,"rx":1462,"fill":85,"stroke":101,"style":87},[64,1526,1527],{"x":66,"y":1517,"fill":68,"style":91},"Mid tier",[64,1529,1530],{"x":66,"y":1521,"fill":75,"style":241},"$3 in \u002F $15 out",[57,1532],{"x":1497,"y":1507,"width":112,"height":1514,"rx":1462,"fill":85,"stroke":86,"style":87},[64,1534,1535],{"x":1475,"y":1517,"fill":68,"style":91},"Frontier tier",[64,1537,1538],{"x":1475,"y":1521,"fill":75,"style":241},"$5 in \u002F $25 out",[64,1540,1541],{"x":66,"y":280,"fill":75,"style":150},"Promote a task only after the cheap tier measurably fails",[283,1543,1545],{"className":285,"code":1544,"language":287,"meta":288,"style":288},"import os\nfrom dataclasses import dataclass\n\n\n@dataclass(frozen=True, slots=True)\nclass Tier:\n    model: str\n    input_per_mtok: float\n    output_per_mtok: float\n    max_tokens: int\n\n\nTIERS = {\n    \"small\": Tier(os.getenv(\"LLM_TIER_SMALL\", \"claude-haiku-4-5\"), 1.00, 5.00, 512),\n    \"mid\": Tier(os.getenv(\"LLM_TIER_MID\", \"claude-sonnet-5\"), 3.00, 15.00, 2048),\n    \"frontier\": Tier(os.getenv(\"LLM_TIER_TOP\", \"claude-opus-4-8\"), 5.00, 25.00, 4096),\n}\n\n\ndef route(task: str, input_tokens: int) -> Tier:\n    match task:\n        case \"classify\" | \"extract\" | \"tag\" | \"route\":\n            return TIERS[\"small\"]\n        case \"summarize\" if input_tokens \u003C 8_000:\n            return TIERS[\"small\"]\n        case \"summarize\" | \"draft\" | \"rewrite\":\n            return TIERS[\"mid\"]\n        case _:\n            return TIERS[\"frontier\"]\n",[163,1546,1547,1553,1563,1567,1571,1593,1602,1609,1617,1624,1631,1635,1639,1649,1682,1714,1745,1750,1754,1758,1779,1787,1813,1830,1851,1863,1881,1894,1901],{"__ignoreMap":288},[292,1548,1549,1551],{"class":229,"line":294},[292,1550,298],{"class":297},[292,1552,302],{"class":301},[292,1554,1555,1557,1559,1561],{"class":229,"line":305},[292,1556,308],{"class":297},[292,1558,311],{"class":301},[292,1560,298],{"class":297},[292,1562,316],{"class":301},[292,1564,1565],{"class":229,"line":319},[292,1566,323],{"emptyLinePlaceholder":322},[292,1568,1569],{"class":229,"line":326},[292,1570,323],{"emptyLinePlaceholder":322},[292,1572,1573,1575,1577,1579,1581,1583,1585,1587,1589,1591],{"class":229,"line":334},[292,1574,522],{"class":490},[292,1576,494],{"class":301},[292,1578,528],{"class":527},[292,1580,345],{"class":297},[292,1582,533],{"class":359},[292,1584,373],{"class":301},[292,1586,538],{"class":527},[292,1588,345],{"class":297},[292,1590,533],{"class":359},[292,1592,379],{"class":301},[292,1594,1595,1597,1600],{"class":229,"line":339},[292,1596,487],{"class":297},[292,1598,1599],{"class":490}," Tier",[292,1601,555],{"class":301},[292,1603,1604,1607],{"class":229,"line":351},[292,1605,1606],{"class":301},"    model: ",[292,1608,564],{"class":359},[292,1610,1611,1614],{"class":229,"line":356},[292,1612,1613],{"class":301},"    input_per_mtok: ",[292,1615,1616],{"class":359},"float\n",[292,1618,1619,1622],{"class":229,"line":382},[292,1620,1621],{"class":301},"    output_per_mtok: ",[292,1623,1616],{"class":359},[292,1625,1626,1629],{"class":229,"line":407},[292,1627,1628],{"class":301},"    max_tokens: ",[292,1630,573],{"class":359},[292,1632,1633],{"class":229,"line":429},[292,1634,323],{"emptyLinePlaceholder":322},[292,1636,1637],{"class":229,"line":452},[292,1638,323],{"emptyLinePlaceholder":322},[292,1640,1641,1644,1646],{"class":229,"line":474},[292,1642,1643],{"class":359},"TIERS",[292,1645,363],{"class":297},[292,1647,1648],{"class":301}," {\n",[292,1650,1651,1654,1657,1660,1662,1664,1667,1670,1672,1675,1677,1680],{"class":229,"line":479},[292,1652,1653],{"class":369},"    \"small\"",[292,1655,1656],{"class":301},": Tier(os.getenv(",[292,1658,1659],{"class":369},"\"LLM_TIER_SMALL\"",[292,1661,373],{"class":301},[292,1663,376],{"class":369},[292,1665,1666],{"class":301},"), ",[292,1668,1669],{"class":359},"1.00",[292,1671,373],{"class":301},[292,1673,1674],{"class":359},"5.00",[292,1676,373],{"class":301},[292,1678,1679],{"class":359},"512",[292,1681,955],{"class":301},[292,1683,1684,1687,1689,1692,1694,1697,1699,1702,1704,1707,1709,1712],{"class":229,"line":484},[292,1685,1686],{"class":369},"    \"mid\"",[292,1688,1656],{"class":301},[292,1690,1691],{"class":369},"\"LLM_TIER_MID\"",[292,1693,373],{"class":301},[292,1695,1696],{"class":369},"\"claude-sonnet-5\"",[292,1698,1666],{"class":301},[292,1700,1701],{"class":359},"3.00",[292,1703,373],{"class":301},[292,1705,1706],{"class":359},"15.00",[292,1708,373],{"class":301},[292,1710,1711],{"class":359},"2048",[292,1713,955],{"class":301},[292,1715,1716,1719,1721,1724,1726,1729,1731,1733,1735,1738,1740,1743],{"class":229,"line":503},[292,1717,1718],{"class":369},"    \"frontier\"",[292,1720,1656],{"class":301},[292,1722,1723],{"class":369},"\"LLM_TIER_TOP\"",[292,1725,373],{"class":301},[292,1727,1728],{"class":369},"\"claude-opus-4-8\"",[292,1730,1666],{"class":301},[292,1732,1674],{"class":359},[292,1734,373],{"class":301},[292,1736,1737],{"class":359},"25.00",[292,1739,373],{"class":301},[292,1741,1742],{"class":359},"4096",[292,1744,955],{"class":301},[292,1746,1747],{"class":229,"line":509},[292,1748,1749],{"class":301},"}\n",[292,1751,1752],{"class":229,"line":514},[292,1753,323],{"emptyLinePlaceholder":322},[292,1755,1756],{"class":229,"line":519},[292,1757,323],{"emptyLinePlaceholder":322},[292,1759,1760,1762,1765,1768,1770,1773,1776],{"class":229,"line":547},[292,1761,1124],{"class":297},[292,1763,1764],{"class":490}," route",[292,1766,1767],{"class":301},"(task: ",[292,1769,693],{"class":359},[292,1771,1772],{"class":301},", input_tokens: ",[292,1774,1775],{"class":359},"int",[292,1777,1778],{"class":301},") -> Tier:\n",[292,1780,1781,1784],{"class":229,"line":558},[292,1782,1783],{"class":297},"    match",[292,1785,1786],{"class":301}," task:\n",[292,1788,1789,1792,1795,1798,1801,1803,1806,1808,1811],{"class":229,"line":567},[292,1790,1791],{"class":297},"        case",[292,1793,1794],{"class":369}," \"classify\"",[292,1796,1797],{"class":297}," |",[292,1799,1800],{"class":369}," \"extract\"",[292,1802,1797],{"class":297},[292,1804,1805],{"class":369}," \"tag\"",[292,1807,1797],{"class":297},[292,1809,1810],{"class":369}," \"route\"",[292,1812,555],{"class":301},[292,1814,1815,1818,1821,1824,1827],{"class":229,"line":576},[292,1816,1817],{"class":297},"            return",[292,1819,1820],{"class":359}," TIERS",[292,1822,1823],{"class":301},"[",[292,1825,1826],{"class":369},"\"small\"",[292,1828,1829],{"class":301},"]\n",[292,1831,1832,1834,1837,1840,1843,1846,1849],{"class":229,"line":584},[292,1833,1791],{"class":297},[292,1835,1836],{"class":369}," \"summarize\"",[292,1838,1839],{"class":297}," if",[292,1841,1842],{"class":301}," input_tokens ",[292,1844,1845],{"class":297},"\u003C",[292,1847,1848],{"class":359}," 8_000",[292,1850,555],{"class":301},[292,1852,1853,1855,1857,1859,1861],{"class":229,"line":589},[292,1854,1817],{"class":297},[292,1856,1820],{"class":359},[292,1858,1823],{"class":301},[292,1860,1826],{"class":369},[292,1862,1829],{"class":301},[292,1864,1865,1867,1869,1871,1874,1876,1879],{"class":229,"line":598},[292,1866,1791],{"class":297},[292,1868,1836],{"class":369},[292,1870,1797],{"class":297},[292,1872,1873],{"class":369}," \"draft\"",[292,1875,1797],{"class":297},[292,1877,1878],{"class":369}," \"rewrite\"",[292,1880,555],{"class":301},[292,1882,1883,1885,1887,1889,1892],{"class":229,"line":615},[292,1884,1817],{"class":297},[292,1886,1820],{"class":359},[292,1888,1823],{"class":301},[292,1890,1891],{"class":369},"\"mid\"",[292,1893,1829],{"class":301},[292,1895,1896,1898],{"class":229,"line":624},[292,1897,1791],{"class":297},[292,1899,1900],{"class":301}," _:\n",[292,1902,1903,1905,1907,1909,1912],{"class":229,"line":639},[292,1904,1817],{"class":297},[292,1906,1820],{"class":359},[292,1908,1823],{"class":301},[292,1910,1911],{"class":369},"\"frontier\"",[292,1913,1829],{"class":301},[14,1915,1916],{},"A table keeps the policy reviewable by whoever owns the margin, which on a side project is you at midnight.",[1918,1919,1920,1936],"table",{},[1921,1922,1923],"thead",{},[1924,1925,1926,1930,1933],"tr",{},[1927,1928,1929],"th",{},"Task shape",[1927,1931,1932],{},"Tier",[1927,1934,1935],{},"Why",[1937,1938,1939,1951,1961,1972],"tbody",{},[1924,1940,1941,1945,1948],{},[1942,1943,1944],"td",{},"Classify, tag, extract",[1942,1946,1947],{},"Small",[1942,1949,1950],{},"Output is a schema, not prose",[1924,1952,1953,1956,1958],{},[1942,1954,1955],{},"Short summary under 8k",[1942,1957,1947],{},[1942,1959,1960],{},"Quality gap is invisible",[1924,1962,1963,1966,1969],{},[1942,1964,1965],{},"Customer-facing draft",[1942,1967,1968],{},"Mid",[1942,1970,1971],{},"Tone matters, volume is lower",[1924,1973,1974,1977,1980],{},[1942,1975,1976],{},"Multi-step or tool chain",[1942,1978,1979],{},"Frontier",[1942,1981,1982],{},"Errors compound across steps",[27,1984,1986],{"id":1985},"per-customer-spend-caps","Per-customer spend caps",[14,1988,1989],{},"Token budgets protect you from one bad request. Spend caps protect you from one bad customer. Without them, a single account on a flat plan consumes the margin of a hundred others and you find out from the invoice. The pattern is a Redis counter per customer per billing period, incremented atomically before the call with an estimate and reconciled afterwards with actual usage.",[14,1991,1992,1993,1997],{},"Reserve before you spend, not after. If you increment only on completion, a burst of fifty concurrent jobs all read the same under-cap value and all proceed. A Lua script makes check-and-increment one atomic operation, which is the difference between a cap and a suggestion. Pair the ledger with durable records by ",[18,1994,1996],{"href":1995},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Ftracking-api-usage-and-analytics\u002Flogging-api-usage-events-to-postgres\u002F","logging API usage events to Postgres",", so billing disputes have an audit trail Redis cannot give you.",[38,1999,47,2004,47,2007,47,2010,47,2013,47,2020,47,2023,47,2026,47,2029,47,2033,47,2036,47,2039,47,2042,47,2045,47,2049,47,2052,47,2055,47,2059,47,2062,47,2066,47,2070,47,2074,47,2077,47,2081,47,2085,47,2089,47,2092],{"viewBox":2000,"role":41,"ariaLabelledBy":2001,"xmlns":45,"style":46},"0 0 740 270",[2002,2003],"llmcost-cap-t","llmcost-cap-d",[49,2005,2006],{"id":2002},"State machine for a per-customer spend cap",[53,2008,2009],{"id":2003},"A customer moves from under cap to a soft cap at eighty percent where requests are tiered down, then to a hard cap where requests are refused, and finally back to under cap when the billing period resets.",[57,2011],{"x":59,"y":59,"width":73,"height":2012,"fill":62},"270",[184,2014,186,2015,47],{},[188,2016,196,2018,186],{"id":2017,"viewBox":191,"refX":192,"refY":193,"markerWidth":194,"markerHeight":194,"orient":195},"llmcost-cap-arrow",[198,2019],{"d":200,"fill":75},[64,2021,2022],{"x":219,"y":67,"fill":68,"style":69},"One customer, one ledger, three states",[57,2024],{"x":67,"y":1469,"width":207,"height":81,"rx":2025,"fill":85,"stroke":114,"style":87},"10",[64,2027,2028],{"x":80,"y":1388,"fill":68,"style":91},"Under cap",[64,2030,2032],{"x":80,"y":2031,"fill":75,"style":241},"123","serve normally",[57,2034],{"x":2035,"y":1469,"width":207,"height":81,"rx":2025,"fill":85,"stroke":101,"style":87},"220",[64,2037,2038],{"x":142,"y":1388,"fill":68,"style":91},"Soft cap",[64,2040,2041],{"x":142,"y":2031,"fill":75,"style":241},"warn, tier down",[57,2043],{"x":2044,"y":1469,"width":207,"height":81,"rx":2025,"fill":85,"stroke":86,"style":87},"410",[64,2046,2048],{"x":2047,"y":1388,"fill":68,"style":91},"480","Hard cap",[64,2050,2051],{"x":2047,"y":2031,"fill":75,"style":241},"refuse, upsell",[57,2053],{"x":2054,"y":1469,"width":207,"height":81,"rx":2025,"fill":85,"stroke":127,"style":87},"600",[64,2056,2058],{"x":2057,"y":1388,"fill":68,"style":91},"670","Cycle reset",[64,2060,2061],{"x":2057,"y":2031,"fill":75,"style":241},"ledger cleared",[64,2063,2065],{"x":2064,"y":80,"fill":75,"style":241},"195","80%",[229,2067],{"x1":1485,"y1":240,"x2":2068,"y2":240,"stroke":75,"style":2069},"216","stroke-width:2;marker-end:url(#llmcost-cap-arrow);",[64,2071,2073],{"x":2072,"y":80,"fill":75,"style":241},"385","100%",[229,2075],{"x1":113,"y1":240,"x2":2076,"y2":240,"stroke":75,"style":2069},"406",[64,2078,2080],{"x":2079,"y":80,"fill":75,"style":241},"575","month end",[229,2082],{"x1":2083,"y1":240,"x2":2084,"y2":240,"stroke":75,"style":2069},"550","596",[198,2086],{"d":2087,"fill":2088,"stroke":75,"style":2069},"M670 136 L670 196 L100 196 L100 140","none",[64,2090,2091],{"x":2072,"y":2068,"fill":75,"style":150},"A new billing period clears the counter and re-opens the account",[64,2093,2094],{"x":219,"y":258,"fill":75,"style":150},"Reserve the estimate before the call; reconcile the actual after it",[283,2096,2098],{"className":285,"code":2097,"language":287,"meta":288,"style":288},"import os\n\nRESERVE_SCRIPT = \"\"\"\nlocal spent = tonumber(redis.call('GET', KEYS[1]) or '0')\nlocal cap = tonumber(ARGV[1])\nlocal estimate = tonumber(ARGV[2])\nif spent + estimate > cap then\n  return -1\nend\nredis.call('INCRBYFLOAT', KEYS[1], ARGV[2])\nredis.call('EXPIREAT', KEYS[1], ARGV[3])\nreturn 1\n\"\"\"\n\nDEFAULT_CAP_USD = float(os.getenv(\"LLM_CUSTOMER_CAP_USD\", \"5.00\"))\n\n\nclass SpendCapReached(RuntimeError):\n    \"\"\"The customer has used their allowance for this billing period.\"\"\"\n\n\nasync def reserve(customer_id: str, period: str, estimate_usd: float,\n                  period_end_epoch: int, cap_usd: float = DEFAULT_CAP_USD) -> None:\n    script = r.register_script(RESERVE_SCRIPT)\n    allowed = await script(\n        keys=[f\"spend:{customer_id}:{period}\"],\n        args=[cap_usd, estimate_usd, period_end_epoch],\n    )\n    if allowed == -1:\n        raise SpendCapReached(customer_id)\n\n\nasync def settle(customer_id: str, period: str, estimate_usd: float,\n                 actual_usd: float) -> None:\n    delta = actual_usd - estimate_usd\n    if delta:\n        await r.incrbyfloat(f\"spend:{customer_id}:{period}\", delta)\n",[163,2099,2100,2106,2110,2120,2125,2130,2135,2140,2145,2150,2155,2160,2165,2170,2174,2194,2198,2202,2215,2220,2224,2228,2254,2278,2292,2304,2341,2351,2355,2372,2379,2383,2387,2410,2423,2439,2446],{"__ignoreMap":288},[292,2101,2102,2104],{"class":229,"line":294},[292,2103,298],{"class":297},[292,2105,302],{"class":301},[292,2107,2108],{"class":229,"line":305},[292,2109,323],{"emptyLinePlaceholder":322},[292,2111,2112,2115,2117],{"class":229,"line":319},[292,2113,2114],{"class":359},"RESERVE_SCRIPT",[292,2116,363],{"class":297},[292,2118,2119],{"class":369}," \"\"\"\n",[292,2121,2122],{"class":229,"line":326},[292,2123,2124],{"class":369},"local spent = tonumber(redis.call('GET', KEYS[1]) or '0')\n",[292,2126,2127],{"class":229,"line":334},[292,2128,2129],{"class":369},"local cap = tonumber(ARGV[1])\n",[292,2131,2132],{"class":229,"line":339},[292,2133,2134],{"class":369},"local estimate = tonumber(ARGV[2])\n",[292,2136,2137],{"class":229,"line":351},[292,2138,2139],{"class":369},"if spent + estimate > cap then\n",[292,2141,2142],{"class":229,"line":356},[292,2143,2144],{"class":369},"  return -1\n",[292,2146,2147],{"class":229,"line":382},[292,2148,2149],{"class":369},"end\n",[292,2151,2152],{"class":229,"line":407},[292,2153,2154],{"class":369},"redis.call('INCRBYFLOAT', KEYS[1], ARGV[2])\n",[292,2156,2157],{"class":229,"line":429},[292,2158,2159],{"class":369},"redis.call('EXPIREAT', KEYS[1], ARGV[3])\n",[292,2161,2162],{"class":229,"line":452},[292,2163,2164],{"class":369},"return 1\n",[292,2166,2167],{"class":229,"line":474},[292,2168,2169],{"class":369},"\"\"\"\n",[292,2171,2172],{"class":229,"line":479},[292,2173,323],{"emptyLinePlaceholder":322},[292,2175,2176,2179,2181,2183,2185,2188,2190,2192],{"class":229,"line":484},[292,2177,2178],{"class":359},"DEFAULT_CAP_USD",[292,2180,363],{"class":297},[292,2182,437],{"class":359},[292,2184,393],{"class":301},[292,2186,2187],{"class":369},"\"LLM_CUSTOMER_CAP_USD\"",[292,2189,373],{"class":301},[292,2191,469],{"class":369},[292,2193,404],{"class":301},[292,2195,2196],{"class":229,"line":503},[292,2197,323],{"emptyLinePlaceholder":322},[292,2199,2200],{"class":229,"line":509},[292,2201,323],{"emptyLinePlaceholder":322},[292,2203,2204,2206,2209,2211,2213],{"class":229,"line":514},[292,2205,487],{"class":297},[292,2207,2208],{"class":490}," SpendCapReached",[292,2210,494],{"class":301},[292,2212,497],{"class":359},[292,2214,500],{"class":301},[292,2216,2217],{"class":229,"line":519},[292,2218,2219],{"class":369},"    \"\"\"The customer has used their allowance for this billing period.\"\"\"\n",[292,2221,2222],{"class":229,"line":547},[292,2223,323],{"emptyLinePlaceholder":322},[292,2225,2226],{"class":229,"line":558},[292,2227,323],{"emptyLinePlaceholder":322},[292,2229,2230,2232,2234,2237,2240,2242,2245,2247,2250,2252],{"class":229,"line":567},[292,2231,681],{"class":297},[292,2233,684],{"class":297},[292,2235,2236],{"class":490}," reserve",[292,2238,2239],{"class":301},"(customer_id: ",[292,2241,693],{"class":359},[292,2243,2244],{"class":301},", period: ",[292,2246,693],{"class":359},[292,2248,2249],{"class":301},", estimate_usd: ",[292,2251,610],{"class":359},[292,2253,866],{"class":301},[292,2255,2256,2259,2261,2264,2266,2268,2271,2273,2276],{"class":229,"line":576},[292,2257,2258],{"class":301},"                  period_end_epoch: ",[292,2260,1775],{"class":359},[292,2262,2263],{"class":301},", cap_usd: ",[292,2265,610],{"class":359},[292,2267,363],{"class":297},[292,2269,2270],{"class":359}," DEFAULT_CAP_USD",[292,2272,1138],{"class":301},[292,2274,2275],{"class":359},"None",[292,2277,555],{"class":301},[292,2279,2280,2283,2285,2288,2290],{"class":229,"line":584},[292,2281,2282],{"class":301},"    script ",[292,2284,345],{"class":297},[292,2286,2287],{"class":301}," r.register_script(",[292,2289,2114],{"class":359},[292,2291,379],{"class":301},[292,2293,2294,2297,2299,2301],{"class":229,"line":589},[292,2295,2296],{"class":301},"    allowed ",[292,2298,345],{"class":297},[292,2300,740],{"class":297},[292,2302,2303],{"class":301}," script(\n",[292,2305,2306,2309,2311,2313,2316,2319,2321,2324,2326,2329,2331,2334,2336,2338],{"class":229,"line":598},[292,2307,2308],{"class":527},"        keys",[292,2310,345],{"class":297},[292,2312,1823],{"class":301},[292,2314,2315],{"class":297},"f",[292,2317,2318],{"class":369},"\"spend:",[292,2320,815],{"class":359},[292,2322,2323],{"class":301},"customer_id",[292,2325,821],{"class":359},[292,2327,2328],{"class":369},":",[292,2330,815],{"class":359},[292,2332,2333],{"class":301},"period",[292,2335,821],{"class":359},[292,2337,812],{"class":369},[292,2339,2340],{"class":301},"],\n",[292,2342,2343,2346,2348],{"class":229,"line":615},[292,2344,2345],{"class":527},"        args",[292,2347,345],{"class":297},[292,2349,2350],{"class":301},"[cap_usd, estimate_usd, period_end_epoch],\n",[292,2352,2353],{"class":229,"line":624},[292,2354,777],{"class":301},[292,2356,2357,2359,2362,2364,2367,2370],{"class":229,"line":639},[292,2358,783],{"class":297},[292,2360,2361],{"class":301}," allowed ",[292,2363,949],{"class":297},[292,2365,2366],{"class":297}," -",[292,2368,2369],{"class":359},"1",[292,2371,555],{"class":301},[292,2373,2374,2376],{"class":229,"line":656},[292,2375,800],{"class":297},[292,2377,2378],{"class":301}," SpendCapReached(customer_id)\n",[292,2380,2381],{"class":229,"line":668},[292,2382,323],{"emptyLinePlaceholder":322},[292,2384,2385],{"class":229,"line":673},[292,2386,323],{"emptyLinePlaceholder":322},[292,2388,2389,2391,2393,2396,2398,2400,2402,2404,2406,2408],{"class":229,"line":678},[292,2390,681],{"class":297},[292,2392,684],{"class":297},[292,2394,2395],{"class":490}," settle",[292,2397,2239],{"class":301},[292,2399,693],{"class":359},[292,2401,2244],{"class":301},[292,2403,693],{"class":359},[292,2405,2249],{"class":301},[292,2407,610],{"class":359},[292,2409,866],{"class":301},[292,2411,2412,2415,2417,2419,2421],{"class":229,"line":704},[292,2413,2414],{"class":301},"                 actual_usd: ",[292,2416,610],{"class":359},[292,2418,1138],{"class":301},[292,2420,2275],{"class":359},[292,2422,555],{"class":301},[292,2424,2425,2428,2430,2433,2436],{"class":229,"line":732},[292,2426,2427],{"class":301},"    delta ",[292,2429,345],{"class":297},[292,2431,2432],{"class":301}," actual_usd ",[292,2434,2435],{"class":297},"-",[292,2437,2438],{"class":301}," estimate_usd\n",[292,2440,2441,2443],{"class":229,"line":746},[292,2442,783],{"class":297},[292,2444,2445],{"class":301}," delta:\n",[292,2447,2448,2451,2454,2456,2458,2460,2462,2464,2466,2468,2470,2472,2474],{"class":229,"line":774},[292,2449,2450],{"class":297},"        await",[292,2452,2453],{"class":301}," r.incrbyfloat(",[292,2455,2315],{"class":297},[292,2457,2318],{"class":369},[292,2459,815],{"class":359},[292,2461,2323],{"class":301},[292,2463,821],{"class":359},[292,2465,2328],{"class":369},[292,2467,815],{"class":359},[292,2469,2333],{"class":301},[292,2471,821],{"class":359},[292,2473,812],{"class":369},[292,2475,2476],{"class":301},", delta)\n",[14,2478,2479,2480,2484],{},"Set the estimate slightly high: over-reserving costs a customer a few cents of headroom, while under-reserving costs you real money on every concurrent burst. If your plans already meter usage through ",[18,2481,2483],{"href":2482},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fstripe-metered-billing-configuration\u002F","Stripe metered billing configuration",", the same period key feeds both the cap and the invoice.",[27,2486,2488],{"id":2487},"alerting-on-cost-regressions","Alerting on cost regressions",[14,2490,2491],{},"The metric to alert on is not total spend, which rises when you grow — the outcome you want. Alert on cost per successful job, computed on a rolling window against the previous one. A prompt edit that adds 800 tokens of instructions, a retry loop that silently doubles calls, or a routing bug that sends classification work to the frontier tier all show up there within minutes and nowhere else until the invoice.",[14,2493,2494,2495,2499],{},"Emit one structured log line per model call carrying customer ID, task, tier, both token counts, and the dollar figure, then aggregate. ",[18,2496,2498],{"href":2497},"\u002Fscaling-and-operating-production-python-apis\u002Fmonitoring-and-logging-python-apis\u002Fstructured-logging-with-structlog\u002F","Structured logging with structlog"," makes those lines queryable without regex archaeology. Set the threshold at a 25% rise in median cost per job over the trailing hour against the same hour a day earlier — tight enough to catch a bad deploy, loose enough to survive one customer with long documents.",[27,2501,2503],{"id":2502},"when-to-use-each-control-and-when-not-to","When to use each control, and when not to",[14,2505,2506],{},"At the prototype stage, ship token budgets and nothing else. They take twenty minutes and stop the catastrophic case. Deduplication earns its keep once you have repeat traffic — below a few thousand jobs a month, the Redis instance costs more than the calls it saves. Prompt caching pays from the second identical-prefix request onward, so turn it on as soon as your system prompt stabilises above about a thousand tokens; below that, most providers will not cache the prefix at all.",[14,2508,2509,2510,1005],{},"Model tiering is worth the routing table once your monthly model bill passes the cost of a day of your own time. Per-customer spend caps become mandatory the moment you sell a flat-rate plan with model calls behind it. Skip them if you charge purely per token with a healthy markup — you have no exposure, and the enforcement code is pure complexity. The exception is free tiers, where caps double as the mechanism for ",[18,2511,2513],{"href":2512},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fpreventing-free-tier-abuse\u002F","preventing free-tier abuse",[27,2515,2517],{"id":2516},"migration-path-from-an-uncontrolled-service","Migration path from an uncontrolled service",[14,2519,2520],{},"Adding all of these at once to a running service is how you ship an outage. Do it in this order, one deploy each.",[2522,2523,2524,2528,2534,2537,2540,2543],"ol",{},[2525,2526,2527],"li",{},"Add the cost ledger in observe-only mode. Log tokens and dollars per call for a week before changing any behaviour.",[2525,2529,2530,2531,2533],{},"Add ",[163,2532,165],{}," and the input count check, but log violations instead of raising. The log shows where a real cap would fire, and on whose traffic.",[2525,2535,2536],{},"Flip the input check to raising, with the limit just above the 99th percentile you measured. The tail that breaks was always the expensive tail.",[2525,2538,2539],{},"Introduce the fingerprint cache with a one-hour TTL to prove the hit rate, then extend it once you trust the fingerprint.",[2525,2541,2542],{},"Add the routing table, moving one task type down a tier at a time and diffing outputs against saved samples before the next move.",[2525,2544,2545],{},"Add spend caps last: a warning-only soft cap first, then refusal once you have a week of real distributions.",[27,2547,2549],{"id":2548},"builder-verdict","Builder verdict",[14,2551,2552,2553,2557],{},"If you ship only two of these, ship the per-request token budget and the fingerprint cache. Together they take an afternoon, need no infrastructure beyond the Redis you already run for ",[18,2554,2556],{"href":2555},"\u002Fscaling-and-operating-production-python-apis\u002Fcaching-python-api-responses-with-redis\u002F","caching Python API responses with Redis",", and they remove both the catastrophic tail and the dumbest recurring spend. Model tiering saves more in absolute dollars, but it costs you an evaluation harness and ongoing judgement calls about quality — real engineering time you might rather spend on features that grow revenue. Per-customer spend caps stop being optional the day you sell a flat-rate plan, not because they save the most but because they are the only thing between one enthusiastic customer and a month of negative margin. Build the ledger first: every other control is guesswork until you know what a job costs.",[27,2559,2561],{"id":2560},"faq","FAQ",[14,2563,2564,2568],{},[2565,2566,2567],"strong",{},"How much of my revenue should model spend be?","\nKeep model spend under 25% of the revenue from the feature it powers, and treat 40% as the line where the product needs repricing rather than more optimisation. On a nineteen-dollar plan that is roughly five dollars of calls a month — about twelve hundred typical jobs on the small tier. If many accounts run past that, the fix is a usage-based plan, not a cleverer cache.",[14,2570,2571,2574],{},[2565,2572,2573],{},"Will a cheaper model cost me customers?","\nOnly if you route by guess instead of by evaluation. Save fifty real production inputs with their frontier-tier outputs, run the cheap tier on the same inputs, and diff. On classification and extraction the results are usually indistinguishable, and the tasks where they are not become obvious immediately. Never testing is the worse risk: you pay five times over for quality nobody perceives.",[14,2576,2577,2580],{},[2565,2578,2579],{},"What happens to caching when I change my prompt?","\nEvery answer keyed on the old prompt version is orphaned, which is what you want — bump the version constant in the fingerprint and old entries expire on their own TTL. Provider-side caching also resets, so the first request after a deploy pays the cache-write premium of about 1.25 times the normal input rate. Budget one cold hour and never ship prompt edits at your traffic peak.",[14,2582,2583,2586],{},[2565,2584,2585],{},"Do spend caps hurt conversion?","\nA hard cap with no warning does. A soft cap that emails the customer at 80%, quietly routes them to a cheaper tier, and offers a one-click upgrade converts better than an unlimited plan you silently rate-limit. Treat the cap as a pricing surface, not a defence mechanism — the customers who hit it get the most value and are your easiest upsell.",[14,2588,2589,2592,2593,2595,2596,2598,2599,2603],{},[2565,2590,2591],{},"How do I estimate cost before the job runs, for a reservation?","\nCount the input tokens exactly, then assume the output hits your configured ",[163,2594,165],{},". That over-reserves on most calls and reconciles down at settlement, which is the safe direction. For streaming endpoints, where output length is unknown until the stream closes, reserve on ",[163,2597,165],{}," up front and settle when the final usage arrives — ",[18,2600,2602],{"href":2601},"\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fstreaming-llm-responses-through-fastapi\u002F","streaming LLM responses through FastAPI"," covers where to hook that settlement.",[27,2605,2607],{"id":2606},"related","Related",[2609,2610,2611,2616,2622,2628,2634],"ul",{},[2525,2612,2613,2615],{},[18,2614,21],{"href":20}," — the parent guide, where the queue worker these controls wrap gets built.",[2525,2617,2618,2621],{},[18,2619,2620],{"href":2601},"Streaming LLM Responses Through FastAPI"," — settling a spend reservation when output length is unknown until the stream ends.",[2525,2623,2624,2627],{},[18,2625,2626],{"href":988},"Calculating Cost per API Request"," — the unit-economics maths behind setting the cap number.",[2525,2629,2630,2633],{},[18,2631,2632],{"href":2555},"Caching Python API Responses with Redis"," — the cache layer the fingerprint deduplication above assumes.",[2525,2635,2636,2640],{},[18,2637,2639],{"href":2638},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Ftracking-api-usage-and-analytics\u002F","Tracking API Usage and Analytics"," — turning the cost ledger into dashboards you and your customers can read.",[2642,2643,2644],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}",{"title":288,"searchDepth":305,"depth":305,"links":2646},[2647,2648,2649,2650,2651,2652,2653,2654,2655,2656,2657],{"id":29,"depth":305,"text":30},{"id":154,"depth":305,"text":155},{"id":993,"depth":305,"text":994},{"id":1433,"depth":305,"text":1434},{"id":1985,"depth":305,"text":1986},{"id":2487,"depth":305,"text":2488},{"id":2502,"depth":305,"text":2503},{"id":2516,"depth":305,"text":2517},{"id":2548,"depth":305,"text":2549},{"id":2560,"depth":305,"text":2561},{"id":2606,"depth":305,"text":2607},"Keep model spend below revenue: per-request token budgets, prompt caching and dedup, model tiering, per-customer spend caps, and cost regression alerts.","md",{"pageTitle":2661,"type":2662,"datePublished":2663,"dateModified":2663},"Controlling LLM API Costs in Production Python Services","article","2026-07-23","\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fcontrolling-llm-api-costs-in-production",{"title":5,"description":2658},"automating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fcontrolling-llm-api-costs-in-production\u002Findex","JYBKQSuhJobfg4O0EXJsR1ANns1yIrMi6dxa8y89yl0",{"@context":2669,"@type":2670,"mainEntity":2671},"https:\u002F\u002Fschema.org","FAQPage",[2672,2677,2680,2683,2686],{"@type":2673,"name":2567,"acceptedAnswer":2674},"Question",{"@type":2675,"text":2676},"Answer","Keep model spend under 25% of the revenue from the feature it powers, and treat 40% as the line where the product needs repricing rather than more optimisation. On a nineteen-dollar plan that is roughly five dollars of calls a month — about twelve hundred typical jobs on the small tier. If many accounts run past that, the fix is a usage-based plan, not a cleverer cache.",{"@type":2673,"name":2573,"acceptedAnswer":2678},{"@type":2675,"text":2679},"Only if you route by guess instead of by evaluation. Save fifty real production inputs with their frontier-tier outputs, run the cheap tier on the same inputs, and diff. On classification and extraction the results are usually indistinguishable, and the tasks where they are not become obvious immediately. Never testing is the worse risk: you pay five times over for quality nobody perceives.",{"@type":2673,"name":2579,"acceptedAnswer":2681},{"@type":2675,"text":2682},"Every answer keyed on the old prompt version is orphaned, which is what you want — bump the version constant in the fingerprint and old entries expire on their own TTL. Provider-side caching also resets, so the first request after a deploy pays the cache-write premium of about 1.25 times the normal input rate. Budget one cold hour and never ship prompt edits at your traffic peak.",{"@type":2673,"name":2585,"acceptedAnswer":2684},{"@type":2675,"text":2685},"A hard cap with no warning does. A soft cap that emails the customer at 80%, quietly routes them to a cheaper tier, and offers a one-click upgrade converts better than an unlimited plan you silently rate-limit. Treat the cap as a pricing surface, not a defence mechanism — the customers who hit it get the most value and are your easiest upsell.",{"@type":2673,"name":2591,"acceptedAnswer":2687},{"@type":2675,"text":2688},"Count the input tokens exactly, then assume the output hits your configured max_tokens. That over-reserves on most calls and reconciles down at settlement, which is the safe direction. For streaming endpoints, where output length is unknown until the stream closes, reserve on max_tokens up front and settle when the final usage arrives — streaming LLM responses through FastAPI covers where to hook that settlement.",1784887028358]