[{"data":1,"prerenderedAt":2283},["ShallowReactive",2],{"page-\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fstreaming-llm-responses-through-fastapi\u002F":3,"faq-schema-\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fstreaming-llm-responses-through-fastapi\u002F":2262},{"id":4,"title":5,"body":6,"description":2252,"extension":2253,"meta":2254,"navigation":335,"path":2258,"seo":2259,"stem":2260,"__hash__":2261},"content\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fstreaming-llm-responses-through-fastapi\u002Findex.md","Streaming LLM Responses Through FastAPI",{"type":7,"value":8,"toc":2240},"minimark",[9,13,23,31,34,39,51,62,133,252,270,272,276,279,997,1023,1025,1029,1036,1043,1183,1202,1204,1208,1214,1230,1326,1333,1716,1743,1745,1749,1752,1767,1818,2038,2053,2055,2059,2062,2075,2077,2081,2084,2115,2117,2121,2127,2130,2132,2136,2142,2152,2158,2176,2182,2184,2188,2193,2208,2213,2236],[10,11,5],"h1",{"id":12},"streaming-llm-responses-through-fastapi",[14,15,16,17,22],"p",{},"You already know how to call a model and wait for the whole answer. This page resolves a narrower and more expensive question: when a paying customer hits your endpoint and you are relaying a model's output token by token, what wire format do you use, and what happens to the meter when they close the tab halfway through? Part of the ",[18,19,21],"a",{"href":20},"\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002F","Automating AI Workflows with Python APIs"," guide, which covers the background-worker shape; this page covers the one case where holding an HTTP connection open for ninety seconds is the correct design.",[14,24,25,26,30],{},"The decision is short, so here it is up front: ",[27,28,29],"strong",{},"ship server-sent events",". Chunked plain text is easier to write and it will cost you a support ticket per week the moment a customer puts a proxy, a CDN, or a browser between themselves and your API. SSE gives you framed events, a documented reconnect story, and — the part that actually pays for itself — somewhere to put usage numbers that is not the response body your customer is parsing as prose.",[32,33],"hr",{},[35,36,38],"h2",{"id":37},"sse-or-a-raw-chunked-body","SSE or a raw chunked body",[14,40,41,42,46,47,50],{},"Both options are the same ASGI mechanism underneath: FastAPI's ",[43,44,45],"code",{},"StreamingResponse"," wraps an async generator, and the server flushes each yielded chunk with ",[43,48,49],{},"Transfer-Encoding: chunked",". The difference is entirely in what the bytes mean.",[14,52,53,54,57,58,61],{},"A raw chunked body streams naked text. The client concatenates whatever arrives. There is no way to signal \"this chunk is a usage record, not prose\", no way to mark the end of a logical message without inventing a sentinel, and no standard for what a client should do after a dropped connection. Server-sent events add a two-line framing (",[43,55,56],{},"event:"," and ",[43,59,60],{},"data:",", terminated by a blank line) that costs you roughly twelve bytes per chunk and buys you typed events, JSON payloads, and a client contract every HTTP library on earth already understands.",[63,64,65,81],"table",{},[66,67,68],"thead",{},[69,70,71,75,78],"tr",{},[72,73,74],"th",{},"Concern",[72,76,77],{},"SSE",[72,79,80],{},"Raw chunked text",[82,83,84,100,111,122],"tbody",{},[69,85,86,90,97],{},[87,88,89],"td",{},"Framing",[87,91,92,94,95],{},[43,93,56],{}," + ",[43,96,60],{},[87,98,99],{},"none",[69,101,102,105,108],{},[87,103,104],{},"Metadata mid-stream",[87,106,107],{},"typed events",[87,109,110],{},"sentinel hacks",[69,112,113,116,119],{},[87,114,115],{},"Proxy buffering",[87,117,118],{},"disable per-response",[87,120,121],{},"often silent",[69,123,124,127,130],{},[87,125,126],{},"Overhead per chunk",[87,128,129],{},"~12 bytes",[87,131,132],{},"0",[134,135,143,144,143,148,143,152,143,158,143,166,143,172,143,180,143,185,143,188,143,192,143,196,143,200,143,205,143,210,143,214,143,218,143,222,143,227,143,230,143,234,143,238,143,241,143,245,143,249],"svg",{"viewBox":136,"role":137,"ariaLabelledBy":138,"xmlns":141,"style":142},"0 0 740 300","img",[139,140],"sllm-wire-t","sllm-wire-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;height:auto;margin:1.5rem 0;font-family:var(--font-sans);","\n  ",[145,146,147],"title",{"id":139},"Wire format comparison between chunked text and server-sent events",[149,150,151],"desc",{"id":140},"A raw chunked body delivers undifferentiated text fragments, while a server-sent event stream delivers named token, usage, and done events that a client can route separately.",[153,154],"rect",{"x":132,"y":132,"width":155,"height":156,"fill":157},"740","300","var(--c-surface)",[159,160,165],"text",{"x":161,"y":162,"fill":163,"style":164},"370","30","var(--c-text)","text-anchor:middle;font-size:14;font-family:var(--font-sans);","Same bytes on the socket, very different contract",[159,167,171],{"x":168,"y":169,"fill":163,"style":170},"180","62","text-anchor:middle;font-size:13;font-family:var(--font-sans);","Raw chunked body",[153,173],{"x":162,"y":174,"width":156,"height":175,"rx":176,"fill":177,"stroke":178,"style":179},"76","46","8","var(--c-surface-alt)","var(--c-border)","stroke-width:2;",[159,181,184],{"x":168,"y":182,"fill":163,"style":183},"105","text-anchor:middle;font-size:12;font-family:var(--font-sans);","\"The quick brown\"",[153,186],{"x":162,"y":187,"width":156,"height":175,"rx":176,"fill":177,"stroke":178,"style":179},"130",[159,189,191],{"x":168,"y":190,"fill":163,"style":183},"159","\" fox jumps over\"",[153,193],{"x":162,"y":194,"width":156,"height":175,"rx":176,"fill":177,"stroke":195,"style":179},"184","var(--c-coral)",[159,197,199],{"x":168,"y":198,"fill":163,"style":183},"213","\"[[USAGE 812]]\" — a hack",[159,201,204],{"x":168,"y":202,"fill":203,"style":183},"258","var(--c-text-muted)","Client cannot tell prose from metadata",[206,207],"line",{"x1":161,"y1":208,"x2":161,"y2":209,"stroke":178,"style":179},"60","262",[159,211,213],{"x":212,"y":169,"fill":163,"style":170},"560","Server-sent events",[153,215],{"x":216,"y":174,"width":156,"height":175,"rx":176,"fill":157,"stroke":217,"style":179},"410","var(--c-blue)",[159,219,221],{"x":212,"y":220,"fill":163,"style":183},"97","event: token",[159,223,226],{"x":212,"y":224,"fill":203,"style":225},"113","text-anchor:middle;font-size:11;font-family:var(--font-sans);","data: {\"t\": \"The quick brown\"}",[153,228],{"x":216,"y":187,"width":156,"height":175,"rx":176,"fill":157,"stroke":229,"style":179},"var(--c-teal)",[159,231,233],{"x":212,"y":232,"fill":163,"style":183},"151","event: usage",[159,235,237],{"x":212,"y":236,"fill":203,"style":225},"167","data: {\"in\": 412, \"out\": 812}",[153,239],{"x":216,"y":194,"width":156,"height":175,"rx":176,"fill":157,"stroke":240,"style":179},"var(--c-yellow)",[159,242,244],{"x":212,"y":243,"fill":163,"style":183},"205","event: done",[159,246,248],{"x":212,"y":247,"fill":203,"style":225},"221","data: {\"stop\": \"end_turn\"}",[159,250,251],{"x":212,"y":202,"fill":203,"style":183},"Each event routes to a different handler",[14,253,254,255,57,258,261,262,265,266,269],{},"There is one deployment gotcha worth internalising before you write a line of code: intermediaries buffer. Nginx buffers proxied responses by default, several PaaS edge layers buffer anything they think is a document, and a customer's corporate proxy will happily hold your stream until it has 8 KB or the connection closes. You defeat this with ",[43,256,257],{},"X-Accel-Buffering: no",[43,259,260],{},"Cache-Control: no-cache",", which most edges honour for ",[43,263,264],{},"text\u002Fevent-stream"," and ignore for ",[43,267,268],{},"text\u002Fplain",". That alone settles the format argument.",[32,271],{},[35,273,275],{"id":274},"the-proxy-endpoint","The proxy endpoint",[14,277,278],{},"Here is the whole thing. It reads config from the environment, streams from the model with the async SDK, re-frames each delta as an SSE event, and emits a final usage event before closing.",[280,281,286],"pre",{"className":282,"code":283,"language":284,"meta":285,"style":285},"language-python shiki shiki-themes github-light github-dark","import asyncio\nimport json\nimport os\nfrom collections.abc import AsyncIterator\n\nfrom anthropic import AsyncAnthropic\nfrom fastapi import FastAPI, Request\nfrom fastapi.responses import StreamingResponse\nfrom pydantic import BaseModel\n\napp = FastAPI()\nclient = AsyncAnthropic(timeout=float(os.getenv(\"LLM_REQUEST_TIMEOUT_S\", \"90\")))\n\nMODEL = os.getenv(\"LLM_MODEL\", \"claude-opus-4-8\")\nMAX_TOKENS = int(os.getenv(\"LLM_MAX_TOKENS\", \"2048\"))\nFIRST_TOKEN_DEADLINE_S = float(os.getenv(\"LLM_FIRST_TOKEN_DEADLINE_S\", \"20\"))\n\n\nclass Prompt(BaseModel):\n    text: str\n\n\ndef sse(event: str, payload: dict) -> bytes:\n    return f\"event: {event}\\ndata: {json.dumps(payload)}\\n\\n\".encode()\n\n\nasync def relay(prompt: str, request: Request) -> AsyncIterator[bytes]:\n    async with client.messages.stream(\n        model=MODEL,\n        max_tokens=MAX_TOKENS,\n        messages=[{\"role\": \"user\", \"content\": prompt}],\n    ) as stream:\n        async for text in stream.text_stream:\n            if await request.is_disconnected():\n                break\n            yield sse(\"token\", {\"t\": text})\n\n        final = await stream.get_final_message()\n        yield sse(\n            \"usage\",\n            {\n                \"in\": final.usage.input_tokens,\n                \"out\": final.usage.output_tokens,\n                \"stop\": final.stop_reason,\n            },\n        )\n    yield sse(\"done\", {})\n\n\n@app.post(\"\u002Fv1\u002Fcomplete\")\nasync def complete(body: Prompt, request: Request) -> StreamingResponse:\n    return StreamingResponse(\n        relay(body.text, request),\n        media_type=\"text\u002Fevent-stream\",\n        headers={\"Cache-Control\": \"no-cache\", \"X-Accel-Buffering\": \"no\"},\n    )\n","python","",[43,287,288,300,308,316,330,337,350,363,376,389,394,406,443,448,471,495,518,523,528,547,556,561,566,596,634,639,644,669,681,694,706,734,746,764,776,782,803,808,821,830,838,844,853,862,871,877,883,897,902,907,920,933,941,947,960,991],{"__ignoreMap":285},[289,290,292,296],"span",{"class":206,"line":291},1,[289,293,295],{"class":294},"szBVR","import",[289,297,299],{"class":298},"sVt8B"," asyncio\n",[289,301,303,305],{"class":206,"line":302},2,[289,304,295],{"class":294},[289,306,307],{"class":298}," json\n",[289,309,311,313],{"class":206,"line":310},3,[289,312,295],{"class":294},[289,314,315],{"class":298}," os\n",[289,317,319,322,325,327],{"class":206,"line":318},4,[289,320,321],{"class":294},"from",[289,323,324],{"class":298}," collections.abc ",[289,326,295],{"class":294},[289,328,329],{"class":298}," AsyncIterator\n",[289,331,333],{"class":206,"line":332},5,[289,334,336],{"emptyLinePlaceholder":335},true,"\n",[289,338,340,342,345,347],{"class":206,"line":339},6,[289,341,321],{"class":294},[289,343,344],{"class":298}," anthropic ",[289,346,295],{"class":294},[289,348,349],{"class":298}," AsyncAnthropic\n",[289,351,353,355,358,360],{"class":206,"line":352},7,[289,354,321],{"class":294},[289,356,357],{"class":298}," fastapi ",[289,359,295],{"class":294},[289,361,362],{"class":298}," FastAPI, Request\n",[289,364,366,368,371,373],{"class":206,"line":365},8,[289,367,321],{"class":294},[289,369,370],{"class":298}," fastapi.responses ",[289,372,295],{"class":294},[289,374,375],{"class":298}," StreamingResponse\n",[289,377,379,381,384,386],{"class":206,"line":378},9,[289,380,321],{"class":294},[289,382,383],{"class":298}," pydantic ",[289,385,295],{"class":294},[289,387,388],{"class":298}," BaseModel\n",[289,390,392],{"class":206,"line":391},10,[289,393,336],{"emptyLinePlaceholder":335},[289,395,397,400,403],{"class":206,"line":396},11,[289,398,399],{"class":298},"app ",[289,401,402],{"class":294},"=",[289,404,405],{"class":298}," FastAPI()\n",[289,407,409,412,414,417,421,423,427,430,434,437,440],{"class":206,"line":408},12,[289,410,411],{"class":298},"client ",[289,413,402],{"class":294},[289,415,416],{"class":298}," AsyncAnthropic(",[289,418,420],{"class":419},"s4XuR","timeout",[289,422,402],{"class":294},[289,424,426],{"class":425},"sj4cs","float",[289,428,429],{"class":298},"(os.getenv(",[289,431,433],{"class":432},"sZZnC","\"LLM_REQUEST_TIMEOUT_S\"",[289,435,436],{"class":298},", ",[289,438,439],{"class":432},"\"90\"",[289,441,442],{"class":298},")))\n",[289,444,446],{"class":206,"line":445},13,[289,447,336],{"emptyLinePlaceholder":335},[289,449,451,454,457,460,463,465,468],{"class":206,"line":450},14,[289,452,453],{"class":425},"MODEL",[289,455,456],{"class":294}," =",[289,458,459],{"class":298}," os.getenv(",[289,461,462],{"class":432},"\"LLM_MODEL\"",[289,464,436],{"class":298},[289,466,467],{"class":432},"\"claude-opus-4-8\"",[289,469,470],{"class":298},")\n",[289,472,474,477,479,482,484,487,489,492],{"class":206,"line":473},15,[289,475,476],{"class":425},"MAX_TOKENS",[289,478,456],{"class":294},[289,480,481],{"class":425}," int",[289,483,429],{"class":298},[289,485,486],{"class":432},"\"LLM_MAX_TOKENS\"",[289,488,436],{"class":298},[289,490,491],{"class":432},"\"2048\"",[289,493,494],{"class":298},"))\n",[289,496,498,501,503,506,508,511,513,516],{"class":206,"line":497},16,[289,499,500],{"class":425},"FIRST_TOKEN_DEADLINE_S",[289,502,456],{"class":294},[289,504,505],{"class":425}," float",[289,507,429],{"class":298},[289,509,510],{"class":432},"\"LLM_FIRST_TOKEN_DEADLINE_S\"",[289,512,436],{"class":298},[289,514,515],{"class":432},"\"20\"",[289,517,494],{"class":298},[289,519,521],{"class":206,"line":520},17,[289,522,336],{"emptyLinePlaceholder":335},[289,524,526],{"class":206,"line":525},18,[289,527,336],{"emptyLinePlaceholder":335},[289,529,531,534,538,541,544],{"class":206,"line":530},19,[289,532,533],{"class":294},"class",[289,535,537],{"class":536},"sScJk"," Prompt",[289,539,540],{"class":298},"(",[289,542,543],{"class":536},"BaseModel",[289,545,546],{"class":298},"):\n",[289,548,550,553],{"class":206,"line":549},20,[289,551,552],{"class":298},"    text: ",[289,554,555],{"class":425},"str\n",[289,557,559],{"class":206,"line":558},21,[289,560,336],{"emptyLinePlaceholder":335},[289,562,564],{"class":206,"line":563},22,[289,565,336],{"emptyLinePlaceholder":335},[289,567,569,572,575,578,581,584,587,590,593],{"class":206,"line":568},23,[289,570,571],{"class":294},"def",[289,573,574],{"class":536}," sse",[289,576,577],{"class":298},"(event: ",[289,579,580],{"class":425},"str",[289,582,583],{"class":298},", payload: ",[289,585,586],{"class":425},"dict",[289,588,589],{"class":298},") -> ",[289,591,592],{"class":425},"bytes",[289,594,595],{"class":298},":\n",[289,597,599,602,605,608,611,614,617,620,622,625,628,631],{"class":206,"line":598},24,[289,600,601],{"class":294},"    return",[289,603,604],{"class":294}," f",[289,606,607],{"class":432},"\"event: ",[289,609,610],{"class":425},"{",[289,612,613],{"class":298},"event",[289,615,616],{"class":425},"}\\n",[289,618,619],{"class":432},"data: ",[289,621,610],{"class":425},[289,623,624],{"class":298},"json.dumps(payload)",[289,626,627],{"class":425},"}\\n\\n",[289,629,630],{"class":432},"\"",[289,632,633],{"class":298},".encode()\n",[289,635,637],{"class":206,"line":636},25,[289,638,336],{"emptyLinePlaceholder":335},[289,640,642],{"class":206,"line":641},26,[289,643,336],{"emptyLinePlaceholder":335},[289,645,647,650,653,656,659,661,664,666],{"class":206,"line":646},27,[289,648,649],{"class":294},"async",[289,651,652],{"class":294}," def",[289,654,655],{"class":536}," relay",[289,657,658],{"class":298},"(prompt: ",[289,660,580],{"class":425},[289,662,663],{"class":298},", request: Request) -> AsyncIterator[",[289,665,592],{"class":425},[289,667,668],{"class":298},"]:\n",[289,670,672,675,678],{"class":206,"line":671},28,[289,673,674],{"class":294},"    async",[289,676,677],{"class":294}," with",[289,679,680],{"class":298}," client.messages.stream(\n",[289,682,684,687,689,691],{"class":206,"line":683},29,[289,685,686],{"class":419},"        model",[289,688,402],{"class":294},[289,690,453],{"class":425},[289,692,693],{"class":298},",\n",[289,695,697,700,702,704],{"class":206,"line":696},30,[289,698,699],{"class":419},"        max_tokens",[289,701,402],{"class":294},[289,703,476],{"class":425},[289,705,693],{"class":298},[289,707,709,712,714,717,720,723,726,728,731],{"class":206,"line":708},31,[289,710,711],{"class":419},"        messages",[289,713,402],{"class":294},[289,715,716],{"class":298},"[{",[289,718,719],{"class":432},"\"role\"",[289,721,722],{"class":298},": ",[289,724,725],{"class":432},"\"user\"",[289,727,436],{"class":298},[289,729,730],{"class":432},"\"content\"",[289,732,733],{"class":298},": prompt}],\n",[289,735,737,740,743],{"class":206,"line":736},32,[289,738,739],{"class":298},"    ) ",[289,741,742],{"class":294},"as",[289,744,745],{"class":298}," stream:\n",[289,747,749,752,755,758,761],{"class":206,"line":748},33,[289,750,751],{"class":294},"        async",[289,753,754],{"class":294}," for",[289,756,757],{"class":298}," text ",[289,759,760],{"class":294},"in",[289,762,763],{"class":298}," stream.text_stream:\n",[289,765,767,770,773],{"class":206,"line":766},34,[289,768,769],{"class":294},"            if",[289,771,772],{"class":294}," await",[289,774,775],{"class":298}," request.is_disconnected():\n",[289,777,779],{"class":206,"line":778},35,[289,780,781],{"class":294},"                break\n",[289,783,785,788,791,794,797,800],{"class":206,"line":784},36,[289,786,787],{"class":294},"            yield",[289,789,790],{"class":298}," sse(",[289,792,793],{"class":432},"\"token\"",[289,795,796],{"class":298},", {",[289,798,799],{"class":432},"\"t\"",[289,801,802],{"class":298},": text})\n",[289,804,806],{"class":206,"line":805},37,[289,807,336],{"emptyLinePlaceholder":335},[289,809,811,814,816,818],{"class":206,"line":810},38,[289,812,813],{"class":298},"        final ",[289,815,402],{"class":294},[289,817,772],{"class":294},[289,819,820],{"class":298}," stream.get_final_message()\n",[289,822,824,827],{"class":206,"line":823},39,[289,825,826],{"class":294},"        yield",[289,828,829],{"class":298}," sse(\n",[289,831,833,836],{"class":206,"line":832},40,[289,834,835],{"class":432},"            \"usage\"",[289,837,693],{"class":298},[289,839,841],{"class":206,"line":840},41,[289,842,843],{"class":298},"            {\n",[289,845,847,850],{"class":206,"line":846},42,[289,848,849],{"class":432},"                \"in\"",[289,851,852],{"class":298},": final.usage.input_tokens,\n",[289,854,856,859],{"class":206,"line":855},43,[289,857,858],{"class":432},"                \"out\"",[289,860,861],{"class":298},": final.usage.output_tokens,\n",[289,863,865,868],{"class":206,"line":864},44,[289,866,867],{"class":432},"                \"stop\"",[289,869,870],{"class":298},": final.stop_reason,\n",[289,872,874],{"class":206,"line":873},45,[289,875,876],{"class":298},"            },\n",[289,878,880],{"class":206,"line":879},46,[289,881,882],{"class":298},"        )\n",[289,884,886,889,891,894],{"class":206,"line":885},47,[289,887,888],{"class":294},"    yield",[289,890,790],{"class":298},[289,892,893],{"class":432},"\"done\"",[289,895,896],{"class":298},", {})\n",[289,898,900],{"class":206,"line":899},48,[289,901,336],{"emptyLinePlaceholder":335},[289,903,905],{"class":206,"line":904},49,[289,906,336],{"emptyLinePlaceholder":335},[289,908,910,913,915,918],{"class":206,"line":909},50,[289,911,912],{"class":536},"@app.post",[289,914,540],{"class":298},[289,916,917],{"class":432},"\"\u002Fv1\u002Fcomplete\"",[289,919,470],{"class":298},[289,921,923,925,927,930],{"class":206,"line":922},51,[289,924,649],{"class":294},[289,926,652],{"class":294},[289,928,929],{"class":536}," complete",[289,931,932],{"class":298},"(body: Prompt, request: Request) -> StreamingResponse:\n",[289,934,936,938],{"class":206,"line":935},52,[289,937,601],{"class":294},[289,939,940],{"class":298}," StreamingResponse(\n",[289,942,944],{"class":206,"line":943},53,[289,945,946],{"class":298},"        relay(body.text, request),\n",[289,948,950,953,955,958],{"class":206,"line":949},54,[289,951,952],{"class":419},"        media_type",[289,954,402],{"class":294},[289,956,957],{"class":432},"\"text\u002Fevent-stream\"",[289,959,693],{"class":298},[289,961,963,966,968,970,973,975,978,980,983,985,988],{"class":206,"line":962},55,[289,964,965],{"class":419},"        headers",[289,967,402],{"class":294},[289,969,610],{"class":298},[289,971,972],{"class":432},"\"Cache-Control\"",[289,974,722],{"class":298},[289,976,977],{"class":432},"\"no-cache\"",[289,979,436],{"class":298},[289,981,982],{"class":432},"\"X-Accel-Buffering\"",[289,984,722],{"class":298},[289,986,987],{"class":432},"\"no\"",[289,989,990],{"class":298},"},\n",[289,992,994],{"class":206,"line":993},56,[289,995,996],{"class":298},"    )\n",[14,998,999,1000,1003,1004,1007,1008,57,1011,1014,1015,1018,1019,1022],{},"Three details carry the design. ",[43,1001,1002],{},"client.messages.stream"," is an async context manager, so the upstream HTTP connection closes deterministically when the generator exits — including on an exception. ",[43,1005,1006],{},"stream.text_stream"," hands you only the text deltas, which is what you want to relay; the SDK is still tracking the underlying ",[43,1009,1010],{},"message_start",[43,1012,1013],{},"message_delta"," events for you. And ",[43,1016,1017],{},"get_final_message()"," returns the assembled message with populated ",[43,1020,1021],{},"usage",", which is the number you bill on.",[32,1024],{},[35,1026,1028],{"id":1027},"backpressure-and-disconnects","Backpressure and disconnects",[14,1030,1031,1032,1035],{},"Backpressure in an ASGI stream is mostly free and mostly invisible. When you ",[43,1033,1034],{},"yield"," a chunk, Uvicorn awaits the socket write. If the client is reading slowly, that await simply takes longer, your generator is suspended, and no memory piles up. The failure mode is not a buffer explosion — it is a coroutine parked for minutes on a customer whose laptop went to sleep, holding an upstream model connection you are being billed for.",[14,1037,1038,1039,1042],{},"That is why the disconnect check matters, and why the naive version is wrong. ",[43,1040,1041],{},"await request.is_disconnected()"," only reports what the ASGI server already knows, and the server frequently does not know until it tries to write. In practice the check fires one or two chunks late, which is fine. What is not fine is omitting it entirely: without it, your generator keeps pulling deltas from the model until the model finishes, and you pay full output tokens for a response nobody read.",[134,1044,143,1049,143,1052,143,1055,143,1058,143,1076,143,1080,143,1086,143,1091,143,1094,143,1098,143,1102,143,1106,143,1110,143,1114,143,1117,143,1119,143,1121,143,1123,143,1129,143,1134,143,1139,143,1143,143,1148,143,1152,143,1158,143,1162,143,1165,143,1169,143,1172,143,1176,143,1179],{"viewBox":1045,"role":137,"ariaLabelledBy":1046,"xmlns":141,"style":142},"0 0 740 330",[1047,1048],"sllm-drop-t","sllm-drop-d",[145,1050,1051],{"id":1047},"What happens when a customer disconnects mid-stream",[149,1053,1054],{"id":1048},"Sequence showing the browser closing the connection, the ASGI server surfacing the disconnect, the generator breaking out of its loop, and the model connection being closed so no further output tokens are billed.",[153,1056],{"x":132,"y":132,"width":155,"height":1057,"fill":157},"330",[1059,1060,1061,1062,143],"defs",{},"\n    ",[1063,1064,1071,1072,1061],"marker",{"id":1065,"viewBox":1066,"refX":1067,"refY":1068,"markerWidth":1069,"markerHeight":1069,"orient":1070},"sllm-drop-arrow","0 0 10 10","9","5","7","auto-start-reverse","\n      ",[1073,1074],"path",{"d":1075,"fill":203},"M0 0 L10 5 L0 10 z",[159,1077,1079],{"x":161,"y":1078,"fill":163,"style":164},"28","A dropped client, and who stops paying when",[153,1081],{"x":1082,"y":1083,"width":1084,"height":1085,"rx":176,"fill":157,"stroke":217,"style":179},"24","50","140","40",[159,1087,1090],{"x":1088,"y":1089,"fill":163,"style":183},"94","75","Customer",[153,1092],{"x":1093,"y":1083,"width":1084,"height":1085,"rx":176,"fill":157,"stroke":229,"style":179},"204",[159,1095,1097],{"x":1096,"y":1089,"fill":163,"style":183},"274","Uvicorn",[153,1099],{"x":1100,"y":1083,"width":1101,"height":1085,"rx":176,"fill":157,"stroke":240,"style":179},"384","150",[159,1103,1105],{"x":1104,"y":1089,"fill":163,"style":183},"459","Your generator",[153,1107],{"x":1108,"y":1083,"width":1109,"height":1085,"rx":176,"fill":157,"stroke":195,"style":179},"574","142",[159,1111,1113],{"x":1112,"y":1089,"fill":163,"style":183},"645","Model API",[206,1115],{"x1":1088,"y1":1116,"x2":1088,"y2":156,"stroke":178,"style":179},"90",[206,1118],{"x1":1096,"y1":1116,"x2":1096,"y2":156,"stroke":178,"style":179},[206,1120],{"x1":1104,"y1":1116,"x2":1104,"y2":156,"stroke":178,"style":179},[206,1122],{"x1":1112,"y1":1116,"x2":1112,"y2":156,"stroke":178,"style":179},[206,1124],{"x1":1125,"y1":1126,"x2":1127,"y2":1126,"stroke":203,"style":1128},"641","118","463","stroke-width:2;marker-end:url(#sllm-drop-arrow);",[159,1130,1133],{"x":1131,"y":1132,"fill":203,"style":225},"552","112","delta",[206,1135],{"x1":1136,"y1":1137,"x2":1138,"y2":1137,"stroke":203,"style":1128},"455","146","278",[159,1140,1142],{"x":1141,"y":1084,"fill":203,"style":225},"366","yield chunk",[206,1144],{"x1":1145,"y1":1146,"x2":1147,"y2":1146,"stroke":203,"style":1128},"270","174","98",[159,1149,1151],{"x":194,"y":1150,"fill":203,"style":225},"168","socket write",[153,1153],{"x":1154,"y":1155,"width":1156,"height":162,"rx":1157,"fill":177,"stroke":195,"style":179},"34","192","120","6",[159,1159,1161],{"x":1088,"y":1160,"fill":163,"style":225},"212","tab closed",[206,1163],{"x1":1147,"y1":1164,"x2":1145,"y2":1164,"stroke":203,"style":1128},"240",[159,1166,1168],{"x":194,"y":1167,"fill":203,"style":225},"234","write fails",[206,1170],{"x1":1138,"y1":1171,"x2":1136,"y2":1171,"stroke":203,"style":1128},"266",[159,1173,1175],{"x":1141,"y":1174,"fill":203,"style":225},"260","is_disconnected",[206,1177],{"x1":1127,"y1":1178,"x2":1125,"y2":1178,"stroke":203,"style":1128},"292",[159,1180,1182],{"x":1131,"y":1181,"fill":203,"style":225},"286","close: billing stops",[14,1184,1185,1186,1189,1190,1193,1194,1197,1198,1201],{},"There is a second, sharper failure mode. If the ASGI server cancels your task outright — which happens on some deploy targets and on shutdown — the ",[43,1187,1188],{},"async for"," raises a cancellation exception inside your generator. Because the SDK stream is an ",[43,1191,1192],{},"async with"," block, the upstream connection still closes correctly. Do not wrap the whole thing in a bare ",[43,1195,1196],{},"except Exception"," that swallows cancellation; catch ",[43,1199,1200],{},"asyncio.CancelledError"," explicitly if you need to log, then re-raise it.",[32,1203],{},[35,1205,1207],{"id":1206},"token-accounting-mid-stream","Token accounting mid-stream",[14,1209,1210,1211,1213],{},"This is where streaming quietly breaks billing. In a non-streaming call you get one response object with ",[43,1212,1021],{}," attached, you write a row, and you are done. In a stream the accounting arrives in pieces, and the piece you most need shows up last — or, on a disconnect, never.",[14,1215,1216,1217,1219,1220,1223,1224,1226,1227,1229],{},"The underlying event sequence is fixed: ",[43,1218,1010],{}," carries input tokens (you know your cost floor before a single output token exists), a run of ",[43,1221,1222],{},"content_block_delta"," events carry the text, and ",[43,1225,1013],{}," carries the cumulative output token count. The SDK folds these into ",[43,1228,1017],{},", but the shape matters because it tells you exactly what you can salvage from an aborted stream.",[134,1231,143,1236,143,1239,143,1242,143,1245,143,1248,143,1252,143,1256,143,1261,143,1264,143,1269,143,1273,143,1278,143,1283,143,1286,143,1290,143,1294,143,1299,143,1304,143,1308,143,1310,143,1312,143,1314,143,1316,143,1319,143,1323],{"viewBox":1232,"role":137,"ariaLabelledBy":1233,"xmlns":141,"style":142},"0 0 740 280",[1234,1235],"sllm-meter-t","sllm-meter-d",[145,1237,1238],{"id":1234},"Token accounting across the lifetime of one stream",[149,1240,1241],{"id":1235},"Timeline showing input tokens known at message start, output tokens accumulating through content block deltas, and the final billable total arriving with the message delta event.",[153,1243],{"x":132,"y":132,"width":155,"height":1244,"fill":157},"280",[159,1246,1247],{"x":161,"y":1078,"fill":163,"style":164},"What you know, and when you know it",[206,1249],{"x1":1083,"y1":1250,"x2":1251,"y2":1250,"stroke":178,"style":179},"200","700",[153,1253],{"x":1083,"y":1150,"width":1254,"height":1255,"fill":217},"14","32",[153,1257],{"x":1258,"y":1259,"width":1254,"height":1260,"fill":229},"160","152","48",[153,1262],{"x":1145,"y":187,"width":1254,"height":1263,"fill":229},"70",[153,1265],{"x":1266,"y":1267,"width":1254,"height":1268,"fill":229},"380","108","92",[153,1270],{"x":1271,"y":1272,"width":1254,"height":1132,"fill":229},"490","88",[153,1274],{"x":1275,"y":1276,"width":1254,"height":1277,"fill":195},"600","66","134",[159,1279,1282],{"x":1280,"y":1281,"fill":163,"style":225},"57","158","412 in",[159,1284,1285],{"x":236,"y":1109,"fill":163,"style":225},"~90",[159,1287,1289],{"x":1288,"y":1156,"fill":163,"style":225},"277","~240",[159,1291,1293],{"x":1292,"y":1147,"fill":163,"style":225},"387","~470",[159,1295,1298],{"x":1296,"y":1297,"fill":163,"style":225},"497","78","~660",[159,1300,1303],{"x":1301,"y":1302,"fill":163,"style":225},"607","56","812 out",[159,1305,1307],{"x":1280,"y":1306,"fill":203,"style":225},"222","start",[159,1309,1133],{"x":236,"y":1306,"fill":203,"style":225},[159,1311,1133],{"x":1288,"y":1306,"fill":203,"style":225},[159,1313,1133],{"x":1292,"y":1306,"fill":203,"style":225},[159,1315,1133],{"x":1296,"y":1306,"fill":203,"style":225},[159,1317,1318],{"x":1301,"y":1306,"fill":203,"style":225},"final",[159,1320,1322],{"x":161,"y":1321,"fill":203,"style":183},"250","Blue: exact at start. Teal: estimated. Coral: exact, only at the end.",[159,1324,1325],{"x":161,"y":1145,"fill":203,"style":183},"A disconnect before \"final\" leaves you estimating the output side.",[14,1327,1328,1329,1332],{},"The commercially correct pattern is a ",[43,1330,1331],{},"finally"," block that always writes a usage row, flagged with whether the numbers are exact or estimated. Count characters as you relay, divide by roughly four for an English-text estimate, and reconcile against the exact figure whenever the stream completes.",[280,1334,1336],{"className":282,"code":1335,"language":284,"meta":285,"style":285},"import os\nimport time\n\nCHARS_PER_TOKEN = float(os.getenv(\"LLM_CHARS_PER_TOKEN\", \"4.0\"))\n\n\nasync def metered_relay(prompt: str, request: Request, api_key_id: str):\n    started = time.monotonic()\n    chars = 0\n    usage = None\n    try:\n        async with client.messages.stream(\n            model=MODEL,\n            max_tokens=MAX_TOKENS,\n            messages=[{\"role\": \"user\", \"content\": prompt}],\n        ) as stream:\n            async for text in stream.text_stream:\n                if await request.is_disconnected():\n                    break\n                chars += len(text)\n                yield sse(\"token\", {\"t\": text})\n            else:\n                usage = (await stream.get_final_message()).usage\n    finally:\n        match usage:\n            case None:\n                out, exact = int(chars \u002F CHARS_PER_TOKEN), False\n            case _:\n                out, exact = usage.output_tokens, True\n        await record_usage(\n            api_key_id=api_key_id,\n            model=MODEL,\n            output_tokens=out,\n            exact=exact,\n            seconds=time.monotonic() - started,\n        )\n",[43,1337,1338,1344,1351,1355,1376,1380,1384,1404,1414,1424,1434,1441,1449,1460,1471,1492,1501,1514,1523,1528,1542,1557,1564,1580,1587,1595,1605,1629,1636,1648,1656,1666,1676,1686,1696,1712],{"__ignoreMap":285},[289,1339,1340,1342],{"class":206,"line":291},[289,1341,295],{"class":294},[289,1343,315],{"class":298},[289,1345,1346,1348],{"class":206,"line":302},[289,1347,295],{"class":294},[289,1349,1350],{"class":298}," time\n",[289,1352,1353],{"class":206,"line":310},[289,1354,336],{"emptyLinePlaceholder":335},[289,1356,1357,1360,1362,1364,1366,1369,1371,1374],{"class":206,"line":318},[289,1358,1359],{"class":425},"CHARS_PER_TOKEN",[289,1361,456],{"class":294},[289,1363,505],{"class":425},[289,1365,429],{"class":298},[289,1367,1368],{"class":432},"\"LLM_CHARS_PER_TOKEN\"",[289,1370,436],{"class":298},[289,1372,1373],{"class":432},"\"4.0\"",[289,1375,494],{"class":298},[289,1377,1378],{"class":206,"line":332},[289,1379,336],{"emptyLinePlaceholder":335},[289,1381,1382],{"class":206,"line":339},[289,1383,336],{"emptyLinePlaceholder":335},[289,1385,1386,1388,1390,1393,1395,1397,1400,1402],{"class":206,"line":352},[289,1387,649],{"class":294},[289,1389,652],{"class":294},[289,1391,1392],{"class":536}," metered_relay",[289,1394,658],{"class":298},[289,1396,580],{"class":425},[289,1398,1399],{"class":298},", request: Request, api_key_id: ",[289,1401,580],{"class":425},[289,1403,546],{"class":298},[289,1405,1406,1409,1411],{"class":206,"line":365},[289,1407,1408],{"class":298},"    started ",[289,1410,402],{"class":294},[289,1412,1413],{"class":298}," time.monotonic()\n",[289,1415,1416,1419,1421],{"class":206,"line":378},[289,1417,1418],{"class":298},"    chars ",[289,1420,402],{"class":294},[289,1422,1423],{"class":425}," 0\n",[289,1425,1426,1429,1431],{"class":206,"line":391},[289,1427,1428],{"class":298},"    usage ",[289,1430,402],{"class":294},[289,1432,1433],{"class":425}," None\n",[289,1435,1436,1439],{"class":206,"line":396},[289,1437,1438],{"class":294},"    try",[289,1440,595],{"class":298},[289,1442,1443,1445,1447],{"class":206,"line":408},[289,1444,751],{"class":294},[289,1446,677],{"class":294},[289,1448,680],{"class":298},[289,1450,1451,1454,1456,1458],{"class":206,"line":445},[289,1452,1453],{"class":419},"            model",[289,1455,402],{"class":294},[289,1457,453],{"class":425},[289,1459,693],{"class":298},[289,1461,1462,1465,1467,1469],{"class":206,"line":450},[289,1463,1464],{"class":419},"            max_tokens",[289,1466,402],{"class":294},[289,1468,476],{"class":425},[289,1470,693],{"class":298},[289,1472,1473,1476,1478,1480,1482,1484,1486,1488,1490],{"class":206,"line":473},[289,1474,1475],{"class":419},"            messages",[289,1477,402],{"class":294},[289,1479,716],{"class":298},[289,1481,719],{"class":432},[289,1483,722],{"class":298},[289,1485,725],{"class":432},[289,1487,436],{"class":298},[289,1489,730],{"class":432},[289,1491,733],{"class":298},[289,1493,1494,1497,1499],{"class":206,"line":497},[289,1495,1496],{"class":298},"        ) ",[289,1498,742],{"class":294},[289,1500,745],{"class":298},[289,1502,1503,1506,1508,1510,1512],{"class":206,"line":520},[289,1504,1505],{"class":294},"            async",[289,1507,754],{"class":294},[289,1509,757],{"class":298},[289,1511,760],{"class":294},[289,1513,763],{"class":298},[289,1515,1516,1519,1521],{"class":206,"line":525},[289,1517,1518],{"class":294},"                if",[289,1520,772],{"class":294},[289,1522,775],{"class":298},[289,1524,1525],{"class":206,"line":530},[289,1526,1527],{"class":294},"                    break\n",[289,1529,1530,1533,1536,1539],{"class":206,"line":549},[289,1531,1532],{"class":298},"                chars ",[289,1534,1535],{"class":294},"+=",[289,1537,1538],{"class":425}," len",[289,1540,1541],{"class":298},"(text)\n",[289,1543,1544,1547,1549,1551,1553,1555],{"class":206,"line":558},[289,1545,1546],{"class":294},"                yield",[289,1548,790],{"class":298},[289,1550,793],{"class":432},[289,1552,796],{"class":298},[289,1554,799],{"class":432},[289,1556,802],{"class":298},[289,1558,1559,1562],{"class":206,"line":563},[289,1560,1561],{"class":294},"            else",[289,1563,595],{"class":298},[289,1565,1566,1569,1571,1574,1577],{"class":206,"line":568},[289,1567,1568],{"class":298},"                usage ",[289,1570,402],{"class":294},[289,1572,1573],{"class":298}," (",[289,1575,1576],{"class":294},"await",[289,1578,1579],{"class":298}," stream.get_final_message()).usage\n",[289,1581,1582,1585],{"class":206,"line":598},[289,1583,1584],{"class":294},"    finally",[289,1586,595],{"class":298},[289,1588,1589,1592],{"class":206,"line":636},[289,1590,1591],{"class":294},"        match",[289,1593,1594],{"class":298}," usage:\n",[289,1596,1597,1600,1603],{"class":206,"line":641},[289,1598,1599],{"class":294},"            case",[289,1601,1602],{"class":425}," None",[289,1604,595],{"class":298},[289,1606,1607,1610,1612,1614,1617,1620,1623,1626],{"class":206,"line":646},[289,1608,1609],{"class":298},"                out, exact ",[289,1611,402],{"class":294},[289,1613,481],{"class":425},[289,1615,1616],{"class":298},"(chars ",[289,1618,1619],{"class":294},"\u002F",[289,1621,1622],{"class":425}," CHARS_PER_TOKEN",[289,1624,1625],{"class":298},"), ",[289,1627,1628],{"class":425},"False\n",[289,1630,1631,1633],{"class":206,"line":671},[289,1632,1599],{"class":294},[289,1634,1635],{"class":298}," _:\n",[289,1637,1638,1640,1642,1645],{"class":206,"line":683},[289,1639,1609],{"class":298},[289,1641,402],{"class":294},[289,1643,1644],{"class":298}," usage.output_tokens, ",[289,1646,1647],{"class":425},"True\n",[289,1649,1650,1653],{"class":206,"line":696},[289,1651,1652],{"class":294},"        await",[289,1654,1655],{"class":298}," record_usage(\n",[289,1657,1658,1661,1663],{"class":206,"line":708},[289,1659,1660],{"class":419},"            api_key_id",[289,1662,402],{"class":294},[289,1664,1665],{"class":298},"api_key_id,\n",[289,1667,1668,1670,1672,1674],{"class":206,"line":736},[289,1669,1453],{"class":419},[289,1671,402],{"class":294},[289,1673,453],{"class":425},[289,1675,693],{"class":298},[289,1677,1678,1681,1683],{"class":206,"line":748},[289,1679,1680],{"class":419},"            output_tokens",[289,1682,402],{"class":294},[289,1684,1685],{"class":298},"out,\n",[289,1687,1688,1691,1693],{"class":206,"line":766},[289,1689,1690],{"class":419},"            exact",[289,1692,402],{"class":294},[289,1694,1695],{"class":298},"exact,\n",[289,1697,1698,1701,1703,1706,1709],{"class":206,"line":778},[289,1699,1700],{"class":419},"            seconds",[289,1702,402],{"class":294},[289,1704,1705],{"class":298},"time.monotonic() ",[289,1707,1708],{"class":294},"-",[289,1710,1711],{"class":298}," started,\n",[289,1713,1714],{"class":206,"line":784},[289,1715,882],{"class":298},[14,1717,1718,1719,1722,1723,1725,1726,1728,1729,1733,1734,1737,1738,1742],{},"The ",[43,1720,1721],{},"for ... else"," is load-bearing: ",[43,1724,1017],{}," runs only when the loop was never broken, so an abandoned stream skips it rather than blocking on a response the customer no longer wants. Write the row from ",[43,1727,1331],{}," regardless, and feed those rows into the same pipeline you use for ",[18,1730,1732],{"href":1731},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Ftracking-api-usage-and-analytics\u002Flogging-api-usage-events-to-postgres\u002F","logging API usage events to Postgres"," — a ",[43,1735,1736],{},"stripe.billing.MeterEvent"," posted per stream is the cleanest bridge to ",[18,1739,1741],{"href":1740},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fstripe-metered-billing-configuration\u002F","metered billing configuration",".",[32,1744],{},[35,1746,1748],{"id":1747},"timeouts-stack-and-you-need-all-three","Timeouts stack, and you need all three",[14,1750,1751],{},"A single timeout number does not describe a stream. You need three, and they measure different things.",[14,1753,1754,1755,1758,1759,1762,1763,1766],{},"The first is ",[27,1756,1757],{},"time to first token",", the one customers judge you on. Twenty seconds is a generous ceiling; past that, fail loudly rather than hold a socket. The second is ",[27,1760,1761],{},"inter-token idle time"," — a stream that emits nothing for thirty seconds is dead even though its total elapsed time looks healthy. The third is ",[27,1764,1765],{},"total wall clock",", which protects you from a pathological generation that trickles tokens for ten minutes. Neither httpx nor the SDK gives you a wall-clock cap by default: their timeouts are per-read and reset on every byte.",[134,1768,143,1773,143,1776,143,1779,143,1781,143,1784,143,1788,143,1792,143,1795,143,1799,143,1802,143,1806,143,1808,143,1812,143,1815],{"viewBox":1769,"role":137,"ariaLabelledBy":1770,"xmlns":141,"style":142},"0 0 740 250",[1771,1772],"sllm-clock-t","sllm-clock-d",[145,1774,1775],{"id":1771},"Three stacked timeout budgets for one streaming request",[149,1777,1778],{"id":1772},"A first-token deadline of twenty seconds sits inside a thirty-second inter-token idle limit, which sits inside a total wall-clock cap of one hundred and eighty seconds, with the reverse proxy deadline set above all three.",[153,1780],{"x":132,"y":132,"width":155,"height":1321,"fill":157},[159,1782,1783],{"x":161,"y":1078,"fill":163,"style":164},"Three clocks, nested — proxy always outermost",[153,1785],{"x":1085,"y":1260,"width":1786,"height":1787,"rx":176,"fill":177,"stroke":178,"style":179},"660","42",[159,1789,1791],{"x":161,"y":1790,"fill":163,"style":183},"74","Reverse proxy deadline — 240s",[153,1793],{"x":1263,"y":1794,"width":1275,"height":1787,"rx":176,"fill":157,"stroke":195,"style":179},"100",[159,1796,1798],{"x":161,"y":1797,"fill":163,"style":183},"126","Total wall clock — 180s",[153,1800],{"x":1794,"y":1259,"width":1801,"height":1787,"rx":176,"fill":157,"stroke":229,"style":179},"360",[159,1803,1805],{"x":1244,"y":1804,"fill":163,"style":183},"178","Idle between tokens — 30s",[153,1807],{"x":187,"y":1093,"width":1164,"height":1154,"rx":176,"fill":157,"stroke":217,"style":179},[159,1809,1811],{"x":1321,"y":1810,"fill":163,"style":183},"226","First token — 20s",[159,1813,1814],{"x":212,"y":1804,"fill":203,"style":225},"resets on every chunk",[159,1816,1817],{"x":212,"y":1810,"fill":203,"style":225},"fires once, at the start",[280,1819,1821],{"className":282,"code":1820,"language":284,"meta":285,"style":285},"import asyncio\nimport os\n\nIDLE_S = float(os.getenv(\"LLM_IDLE_TIMEOUT_S\", \"30\"))\nTOTAL_S = float(os.getenv(\"LLM_TOTAL_TIMEOUT_S\", \"180\"))\n\n\nasync def guarded(source, request: Request):\n    async with asyncio.timeout(TOTAL_S):\n        iterator = source.__aiter__()\n        while True:\n            try:\n                chunk = await asyncio.wait_for(iterator.__anext__(), IDLE_S)\n            except StopAsyncIteration:\n                return\n            except TimeoutError:\n                yield sse(\"error\", {\"code\": \"upstream_stalled\"})\n                return\n            if await request.is_disconnected():\n                return\n            yield chunk\n",[43,1822,1823,1829,1835,1839,1860,1881,1885,1889,1901,1914,1930,1940,1947,1969,1979,1984,1993,2015,2019,2027,2031],{"__ignoreMap":285},[289,1824,1825,1827],{"class":206,"line":291},[289,1826,295],{"class":294},[289,1828,299],{"class":298},[289,1830,1831,1833],{"class":206,"line":302},[289,1832,295],{"class":294},[289,1834,315],{"class":298},[289,1836,1837],{"class":206,"line":310},[289,1838,336],{"emptyLinePlaceholder":335},[289,1840,1841,1844,1846,1848,1850,1853,1855,1858],{"class":206,"line":318},[289,1842,1843],{"class":425},"IDLE_S",[289,1845,456],{"class":294},[289,1847,505],{"class":425},[289,1849,429],{"class":298},[289,1851,1852],{"class":432},"\"LLM_IDLE_TIMEOUT_S\"",[289,1854,436],{"class":298},[289,1856,1857],{"class":432},"\"30\"",[289,1859,494],{"class":298},[289,1861,1862,1865,1867,1869,1871,1874,1876,1879],{"class":206,"line":332},[289,1863,1864],{"class":425},"TOTAL_S",[289,1866,456],{"class":294},[289,1868,505],{"class":425},[289,1870,429],{"class":298},[289,1872,1873],{"class":432},"\"LLM_TOTAL_TIMEOUT_S\"",[289,1875,436],{"class":298},[289,1877,1878],{"class":432},"\"180\"",[289,1880,494],{"class":298},[289,1882,1883],{"class":206,"line":339},[289,1884,336],{"emptyLinePlaceholder":335},[289,1886,1887],{"class":206,"line":352},[289,1888,336],{"emptyLinePlaceholder":335},[289,1890,1891,1893,1895,1898],{"class":206,"line":365},[289,1892,649],{"class":294},[289,1894,652],{"class":294},[289,1896,1897],{"class":536}," guarded",[289,1899,1900],{"class":298},"(source, request: Request):\n",[289,1902,1903,1905,1907,1910,1912],{"class":206,"line":378},[289,1904,674],{"class":294},[289,1906,677],{"class":294},[289,1908,1909],{"class":298}," asyncio.timeout(",[289,1911,1864],{"class":425},[289,1913,546],{"class":298},[289,1915,1916,1919,1921,1924,1927],{"class":206,"line":391},[289,1917,1918],{"class":298},"        iterator ",[289,1920,402],{"class":294},[289,1922,1923],{"class":298}," source.",[289,1925,1926],{"class":425},"__aiter__",[289,1928,1929],{"class":298},"()\n",[289,1931,1932,1935,1938],{"class":206,"line":396},[289,1933,1934],{"class":294},"        while",[289,1936,1937],{"class":425}," True",[289,1939,595],{"class":298},[289,1941,1942,1945],{"class":206,"line":408},[289,1943,1944],{"class":294},"            try",[289,1946,595],{"class":298},[289,1948,1949,1952,1954,1956,1959,1962,1965,1967],{"class":206,"line":445},[289,1950,1951],{"class":298},"                chunk ",[289,1953,402],{"class":294},[289,1955,772],{"class":294},[289,1957,1958],{"class":298}," asyncio.wait_for(iterator.",[289,1960,1961],{"class":425},"__anext__",[289,1963,1964],{"class":298},"(), ",[289,1966,1843],{"class":425},[289,1968,470],{"class":298},[289,1970,1971,1974,1977],{"class":206,"line":450},[289,1972,1973],{"class":294},"            except",[289,1975,1976],{"class":425}," StopAsyncIteration",[289,1978,595],{"class":298},[289,1980,1981],{"class":206,"line":473},[289,1982,1983],{"class":294},"                return\n",[289,1985,1986,1988,1991],{"class":206,"line":497},[289,1987,1973],{"class":294},[289,1989,1990],{"class":425}," TimeoutError",[289,1992,595],{"class":298},[289,1994,1995,1997,1999,2002,2004,2007,2009,2012],{"class":206,"line":520},[289,1996,1546],{"class":294},[289,1998,790],{"class":298},[289,2000,2001],{"class":432},"\"error\"",[289,2003,796],{"class":298},[289,2005,2006],{"class":432},"\"code\"",[289,2008,722],{"class":298},[289,2010,2011],{"class":432},"\"upstream_stalled\"",[289,2013,2014],{"class":298},"})\n",[289,2016,2017],{"class":206,"line":525},[289,2018,1983],{"class":294},[289,2020,2021,2023,2025],{"class":206,"line":530},[289,2022,769],{"class":294},[289,2024,772],{"class":294},[289,2026,775],{"class":298},[289,2028,2029],{"class":206,"line":549},[289,2030,1983],{"class":294},[289,2032,2033,2035],{"class":206,"line":558},[289,2034,787],{"class":294},[289,2036,2037],{"class":298}," chunk\n",[14,2039,2040,2041,2043,2044,2047,2048,2052],{},"Set the deadline on your reverse proxy higher than ",[43,2042,1864],{},", never lower — a proxy that kills the connection first strips your ability to emit a clean error event, and the customer sees a truncated response instead of a diagnosable failure. If you run Gunicorn in front of Uvicorn workers, raise its ",[43,2045,2046],{},"--timeout"," too; the defaults there are tuned for request-response endpoints, not for the streaming shape, which is one more reason to read the ",[18,2049,2051],{"href":2050},"\u002Fgetting-started-with-python-apis-for-builders\u002Fsetting-up-fastapi\u002Fuvicorn-vs-gunicorn-worker-configuration\u002F","Uvicorn and Gunicorn worker configuration trade-offs"," before you deploy this.",[32,2054],{},[35,2056,2058],{"id":2057},"when-to-stream-and-when-not-to","When to stream, and when not to",[14,2060,2061],{},"Stream when the output is prose a human reads as it appears, when generations routinely exceed five seconds, or when your competitors stream and yours feels broken by comparison. Perceived latency is the entire benefit, and it is worth real money on a chat-shaped product.",[14,2063,2064,2065,2069,2070,2074],{},"Do not stream when the consumer is another program. If your customer parses your output as JSON, streaming buys them nothing and costs you a held connection per request, which is the worst possible concurrency profile on a small box. Do not stream when the work belongs in a queue either — anything over two minutes should be a job with a webhook callback, using the pattern from ",[18,2066,2068],{"href":2067},"\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002F","running background jobs with Celery",". And be careful on serverless platforms billed by wall-clock duration: a ninety-second stream at low concurrency can cost more in compute than the model tokens did, a trap worth modelling with ",[18,2071,2073],{"href":2072},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fcalculating-cost-per-api-request\u002F","cost per API request"," before you commit.",[32,2076],{},[35,2078,2080],{"id":2079},"migrating-from-a-blocking-endpoint","Migrating from a blocking endpoint",[14,2082,2083],{},"If you already have a JSON endpoint that returns the whole completion, the switch is four steps and does not require breaking existing customers.",[2085,2086,2087,2095,2106,2112],"ol",{},[2088,2089,2090,2091,2094],"li",{},"Add a new path — ",[43,2092,2093],{},"\u002Fv1\u002Fcomplete\u002Fstream"," — rather than changing the old one's content type. Existing integrations keep working.",[2088,2096,2097,2098,2101,2102,2105],{},"Move the model call into an async generator, swap ",[43,2099,2100],{},"messages.create"," for ",[43,2103,2104],{},"messages.stream",", and yield SSE frames.",[2088,2107,2108,2109,2111],{},"Move usage recording into a ",[43,2110,1331],{}," block with the exact-versus-estimated flag, so both endpoints feed the same meter.",[2088,2113,2114],{},"Once the streaming path carries the majority of traffic, mark the blocking endpoint deprecated with a sunset header rather than deleting it.",[32,2116],{},[35,2118,2120],{"id":2119},"builder-verdict","Builder verdict",[14,2122,2123,2124,2126],{},"Ship SSE, put the usage numbers in their own event, and write the meter row from a ",[43,2125,1331],{}," block. That combination costs you about forty lines over the blocking version and removes the two failure modes that actually cost money: silent buffering that makes your API look slow, and abandoned streams you either bill wrongly or bill not at all. The twelve bytes of framing per chunk is a rounding error against a single model call.",[14,2128,2129],{},"The one thing worth being strict about is the disconnect check. Every stream you keep pulling after the customer left is pure margin burned — at typical output pricing, one abandoned long generation per hundred requests is a measurable dent in gross margin on a cheap tier. Check on every chunk, break the loop, and let the context manager close the upstream connection. Get that right on day one and streaming becomes a feature you charge for instead of a leak you discover on the invoice.",[32,2131],{},[35,2133,2135],{"id":2134},"faq","FAQ",[14,2137,2138,2141],{},[27,2139,2140],{},"Does streaming cost more than a blocking call?","\nThe model tokens are identical. What changes is your compute bill: you hold a worker slot for the whole generation instead of a fraction of a second, so an async worker that handled 200 requests per second might handle 40 concurrent streams. On a fixed-price VPS that is free capacity you were not using; on per-second serverless it can double your cost per request. Model it before you price the tier.",[14,2143,2144,2147,2148,2151],{},[27,2145,2146],{},"How do I bill a customer who disconnects halfway?","\nBill the estimate and mark it as such. You are charged by the model for every token it generated before you closed the connection, so absorbing that is real margin loss. Record the row with an ",[43,2149,2150],{},"exact=false"," flag, and if disputes appear, credit them — the volume is small and the goodwill is cheaper than the argument.",[14,2153,2154,2157],{},[27,2155,2156],{},"Can I rotate an API key mid-stream?","\nNo, and you should not try. The upstream connection was authenticated when it opened. Rotate keys between requests using an overlap window where both the old and new key validate, then let in-flight streams drain naturally. Anything else drops paying customers mid-sentence.",[14,2159,2160,2163,2164,2166,2167,57,2169,2171,2172,2175],{},[27,2161,2162],{},"What breaks when I put this behind a CDN?","\nBuffering, almost always. A CDN that treats ",[43,2165,264],{}," as a document will hold your chunks until the response completes, turning a streaming API into a slow blocking one with no error to debug. Send ",[43,2168,260],{},[43,2170,257],{},", then verify with ",[43,2173,2174],{},"curl -N"," against the public URL — not against localhost, where the problem never reproduces.",[14,2177,2178,2181],{},[27,2179,2180],{},"Is this migration risky for existing integrations?","\nNot if you add a new path instead of changing the existing one's content type. Customers parsing your JSON endpoint keep parsing it; new customers get the stream. Deprecate the old path only after the traffic has moved, using a sunset header and a dated notice.",[32,2183],{},[35,2185,2187],{"id":2186},"related","Related",[14,2189,2190],{},[27,2191,2192],{},"Same track:",[2194,2195,2196,2201],"ul",{},[2088,2197,2198,2200],{},[18,2199,21],{"href":20}," — the parent guide, covering the background-worker shape this page is the exception to.",[2088,2202,2203,2207],{},[18,2204,2206],{"href":2205},"\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fcontrolling-llm-api-costs-in-production\u002F","Controlling LLM API Costs in Production"," — budget caps and model routing, which is where the usage rows you record here get used.",[14,2209,2210],{},[27,2211,2212],{},"Other tracks:",[2194,2214,2215,2222,2229],{},[2088,2216,2217,2221],{},[18,2218,2220],{"href":2219},"\u002Fgetting-started-with-python-apis-for-builders\u002Fparsing-json-responses\u002Fhandling-large-json-payloads-with-streaming\u002F","Handling Large JSON Payloads with Streaming"," — the same incremental mindset applied to inbound data.",[2088,2223,2224,2228],{},[18,2225,2227],{"href":2226},"\u002Fgetting-started-with-python-apis-for-builders\u002Fsetting-up-fastapi\u002F","Setting Up FastAPI"," — the baseline app structure the endpoint above assumes.",[2088,2230,2231,2235],{},[18,2232,2234],{"href":2233},"\u002Fscaling-and-operating-production-python-apis\u002Fmonitoring-and-logging-python-apis\u002Fstructured-logging-with-structlog\u002F","Structured Logging with structlog"," — how to make disconnect and stall events searchable instead of anecdotal.",[2237,2238,2239],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":285,"searchDepth":302,"depth":302,"links":2241},[2242,2243,2244,2245,2246,2247,2248,2249,2250,2251],{"id":37,"depth":302,"text":38},{"id":274,"depth":302,"text":275},{"id":1027,"depth":302,"text":1028},{"id":1206,"depth":302,"text":1207},{"id":1747,"depth":302,"text":1748},{"id":2057,"depth":302,"text":2058},{"id":2079,"depth":302,"text":2080},{"id":2119,"depth":302,"text":2120},{"id":2134,"depth":302,"text":2135},{"id":2186,"depth":302,"text":2187},"Proxy a streaming model response to your own customers with FastAPI: server-sent events vs chunked bodies, client disconnects, mid-stream token accounting, and timeouts.","md",{"pageTitle":2255,"type":2256,"datePublished":2257,"dateModified":2257},"Streaming LLM Responses Through FastAPI: SSE vs Chunked","article","2026-07-23","\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fstreaming-llm-responses-through-fastapi",{"title":5,"description":2252},"automating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fstreaming-llm-responses-through-fastapi\u002Findex","Io3O1VQt5d_rwWgyVEXam6YdA2XCAePa4NUC-raHU5c",{"@context":2263,"@type":2264,"mainEntity":2265},"https:\u002F\u002Fschema.org","FAQPage",[2266,2271,2274,2277,2280],{"@type":2267,"name":2140,"acceptedAnswer":2268},"Question",{"@type":2269,"text":2270},"Answer","The model tokens are identical. What changes is your compute bill: you hold a worker slot for the whole generation instead of a fraction of a second, so an async worker that handled 200 requests per second might handle 40 concurrent streams. On a fixed-price VPS that is free capacity you were not using; on per-second serverless it can double your cost per request. Model it before you price the tier.",{"@type":2267,"name":2146,"acceptedAnswer":2272},{"@type":2269,"text":2273},"Bill the estimate and mark it as such. You are charged by the model for every token it generated before you closed the connection, so absorbing that is real margin loss. Record the row with an exact=false flag, and if disputes appear, credit them — the volume is small and the goodwill is cheaper than the argument.",{"@type":2267,"name":2156,"acceptedAnswer":2275},{"@type":2269,"text":2276},"No, and you should not try. The upstream connection was authenticated when it opened. Rotate keys between requests using an overlap window where both the old and new key validate, then let in-flight streams drain naturally. Anything else drops paying customers mid-sentence.",{"@type":2267,"name":2162,"acceptedAnswer":2278},{"@type":2269,"text":2279},"Buffering, almost always. A CDN that treats text\u002Fevent-stream as a document will hold your chunks until the response completes, turning a streaming API into a slow blocking one with no error to debug. Send Cache-Control: no-cache and X-Accel-Buffering: no, then verify with curl -N against the public URL — not against localhost, where the problem never reproduces.",{"@type":2267,"name":2180,"acceptedAnswer":2281},{"@type":2269,"text":2282},"Not if you add a new path instead of changing the existing one's content type. Customers parsing your JSON endpoint keep parsing it; new customers get the stream. Deprecate the old path only after the traffic has moved, using a sunset header and a dated notice. ---",1784887028387]