[{"data":1,"prerenderedAt":3047},["ShallowReactive",2],{"page-\u002Fgetting-started-with-python-apis-for-builders\u002Fparsing-json-responses\u002Fhandling-large-json-payloads-with-streaming\u002F":3,"faq-schema-\u002Fgetting-started-with-python-apis-for-builders\u002Fparsing-json-responses\u002Fhandling-large-json-payloads-with-streaming\u002F":3026},{"id":4,"title":5,"body":6,"description":3016,"extension":3017,"meta":3018,"navigation":477,"path":3022,"seo":3023,"stem":3024,"__hash__":3025},"content\u002Fgetting-started-with-python-apis-for-builders\u002Fparsing-json-responses\u002Fhandling-large-json-payloads-with-streaming\u002Findex.md","Handling Large JSON Payloads with Streaming: Parsing 420 MB on a 512 MB Box",{"type":7,"value":8,"toc":3005},"minimark",[9,13,28,39,42,47,60,63,172,174,178,181,240,250,407,429,1169,1177,1546,1553,1555,1559,1562,1921,1946,2043,2045,2049,2061,2607,2619,2720,2740,2742,2746,2754,2760,2858,2860,2864,2874,2886,2888,2892,2908,2912,2919,2925,2935,2944,2950,2954,2959,2980,2985,3001],[10,11,5],"h1",{"id":12},"handling-large-json-payloads-with-streaming-parsing-420-mb-on-a-512-mb-box",[14,15,16,17,21,22,27],"p",{},"There is one line of code that quietly decides whether your import job runs on a $7 instance or a $60 one: ",[18,19,20],"code",{},"json.loads(response.content)",". It works beautifully until a customer sends a 420 MB export, the interpreter allocates roughly seven times that in live Python objects, and your container gets OOM-killed mid-request. This page resolves a single decision — when to abandon whole-document parsing for a streaming parser, which streaming shape to pick, and what measurements prove the switch paid for itself. Part of the ",[23,24,26],"a",{"href":25},"\u002Fgetting-started-with-python-apis-for-builders\u002Fparsing-json-responses\u002F","Parsing JSON Responses"," guide.",[14,29,30,31,34,35,38],{},"The short version: if you control the producer, emit NDJSON and parse it line by line. If you do not, use ",[18,32,33],{},"ijson"," to walk the document as events. Keep ",[18,36,37],{},"json.loads"," for anything under about 50 MB, where its speed advantage is real and its memory cost is irrelevant.",[40,41],"hr",{},[43,44,46],"h2",{"id":45},"why-whole-document-parsing-falls-over","Why Whole-Document Parsing Falls Over",[14,48,49,51,52,55,56,59],{},[18,50,37],{}," has to materialize every object before it returns anything. A 420 MB array of 1.2 million order records becomes 1.2 million ",[18,53,54],{},"dict"," objects, each with its own hash table, plus interned-but-not-free ",[18,57,58],{},"str"," keys and values. Measured on CPython 3.12 with a real Shopify-shaped export, peak resident memory hits about 3.1 GB — a 7.4x expansion over the raw bytes. That figure is not a pathology; it is the normal cost of Python's object model. Every small dict carries around 184 bytes of overhead before you store a single field in it.",[14,61,62],{},"The commercial consequence arrives on your hosting bill. A 512 MB instance costs a few dollars a month; the 4 GB instance you need to survive one customer's export costs roughly ten times that, and you pay for it around the clock to handle a job that runs for eleven seconds a day. Worse, memory ceilings are cliffs rather than slopes. A payload 5% bigger than the one you tested does not run 5% slower — the process dies and the customer retries, doubling the load. Streaming turns that cliff into a flat line: memory stops tracking payload size at all.",[64,65,73,74,73,78,73,82,73,89,73,97,73,105,73,110,73,117,73,123,73,127,73,133,73,138,73,142,73,147,73,152,73,156,73,160,73,163,73,166],"svg",{"viewBox":66,"role":67,"ariaLabelledBy":68,"xmlns":71,"style":72},"0 0 720 250","img",[69,70],"bigjson-mem-t","bigjson-mem-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;height:auto;margin:1.5rem 0;font-family:var(--font-sans);","\n  ",[75,76,77],"title",{"id":69},"Peak memory by parsing strategy",[79,80,81],"desc",{"id":70},"Horizontal bar chart of peak resident memory parsing a 420 MB JSON array: json.loads 3100 MB, ijson stream 48 MB, NDJSON loop 31 MB, NDJSON with orjson 31 MB.",[83,84],"rect",{"x":85,"y":85,"width":86,"height":87,"fill":88},"0","720","250","var(--c-surface)",[90,91,96],"text",{"x":92,"y":93,"fill":94,"style":95},"16","26","var(--c-text)","font-size:13;font-family:var(--font-sans);","Peak RSS parsing a 420 MB JSON array (MB)",[98,99],"line",{"x1":100,"y1":101,"x2":100,"y2":102,"stroke":103,"style":104},"170","42","212","var(--c-border)","stroke-width:1;",[90,106,37],{"x":107,"y":108,"fill":94,"style":109},"160","72","text-anchor:end;font-size:12;font-family:var(--font-sans);",[83,111],{"x":100,"y":112,"width":113,"height":114,"rx":115,"fill":116},"58","436","22","3","var(--c-coral)",[90,118,122],{"x":119,"y":120,"fill":94,"style":121},"614","74","font-size:12;font-family:var(--font-sans);","3100",[90,124,126],{"x":107,"y":125,"fill":94,"style":109},"117","ijson stream",[83,128],{"x":100,"y":129,"width":130,"height":114,"rx":131,"fill":132},"103","7","2","var(--c-blue)",[90,134,137],{"x":135,"y":136,"fill":94,"style":121},"185","119","48",[90,139,141],{"x":107,"y":140,"fill":94,"style":109},"162","NDJSON loop",[83,143],{"x":100,"y":144,"width":145,"height":114,"rx":131,"fill":146},"148","5","var(--c-teal)",[90,148,151],{"x":149,"y":150,"fill":94,"style":121},"183","164","31",[90,153,155],{"x":107,"y":154,"fill":94,"style":109},"207","NDJSON + orjson",[83,157],{"x":100,"y":158,"width":145,"height":114,"rx":131,"fill":159},"193","var(--c-yellow)",[90,161,151],{"x":149,"y":162,"fill":94,"style":121},"209",[98,164],{"x1":100,"y1":102,"x2":165,"y2":102,"stroke":103,"style":104},"690",[90,167,171],{"x":92,"y":168,"fill":169,"style":170},"236","var(--c-text-muted)","font-size:11;font-family:var(--font-sans);","Bars are to scale: the three streaming strategies are hairlines next to the buffered parse.",[40,173],{},[43,175,177],{"id":176},"the-three-streaming-shapes","The Three Streaming Shapes",[14,179,180],{},"Streaming JSON is not one technique. Three distinct shapes cover almost every case a commercial API runs into, and picking the wrong one costs you either throughput or a week of yak-shaving.",[182,183,184,200],"table",{},[185,186,187],"thead",{},[188,189,190,194,197],"tr",{},[191,192,193],"th",{},"Shape",[191,195,196],{},"Memory",[191,198,199],{},"Use when",[201,202,203,216,229],"tbody",{},[188,204,205,210,213],{},[206,207,208],"td",{},[18,209,37],{},[206,211,212],{},"~7x payload",[206,214,215],{},"Payload under 50 MB",[188,217,218,223,226],{},[206,219,220,222],{},[18,221,33],{}," events",[206,224,225],{},"Constant, ~48 MB",[206,227,228],{},"You cannot change the producer",[188,230,231,234,237],{},[206,232,233],{},"NDJSON lines",[206,235,236],{},"Constant, ~31 MB",[206,238,239],{},"You control both ends",[14,241,242,243,245,246,249],{},"The first shape is the baseline you already use. The second, event-based parsing with ",[18,244,33],{},", treats the document as a stream of tokens and yields complete sub-objects at a path you name. It handles any valid JSON, including the single monolithic array that third-party exports always ship. The third changes the format itself: newline-delimited JSON puts one object on each line, so the transport is trivially chunkable and any language can consume it with a ",[18,247,248],{},"for"," loop.",[64,251,73,256,73,259,73,262,73,265,73,281,73,284,73,291,73,297,73,303,73,307,73,312,73,317,73,321,73,323,73,327,73,330,73,334,73,338,73,342,73,346,73,349,73,352,73,356,73,359,73,364,73,368,73,372,73,375,73,378,73,381,73,385,73,387,73,391,73,394,73,399,73,403],{"viewBox":252,"role":67,"ariaLabelledBy":253,"xmlns":71,"style":72},"0 0 720 300",[254,255],"bigjson-flow-t","bigjson-flow-d",[75,257,258],{"id":254},"Buffered parsing versus streaming parsing data flow",[79,260,261],{"id":255},"Two lanes compare a buffered path that accumulates the whole body in RAM before parsing against a streaming path where chunks flow through an event parser to a sink with constant memory.",[83,263],{"x":85,"y":85,"width":86,"height":264,"fill":88},"300",[266,267,268,269,73],"defs",{},"\n    ",[270,271,276,277,268],"marker",{"id":272,"viewBox":273,"refX":274,"refY":145,"markerWidth":130,"markerHeight":130,"orient":275},"bigjson-flow-arrow","0 0 10 10","9","auto-start-reverse","\n      ",[278,279],"path",{"d":280,"fill":169},"M0 0 L10 5 L0 10 z",[90,282,283],{"x":92,"y":93,"fill":94,"style":95},"Buffered: memory grows with the payload",[83,285],{"x":92,"y":101,"width":286,"height":287,"rx":288,"fill":289,"stroke":103,"style":290},"120","46","8","var(--c-surface-alt)","stroke-width:1.5;",[90,292,296],{"x":293,"y":294,"fill":94,"style":295},"76","70","text-anchor:middle;font-size:12;font-family:var(--font-sans);","socket",[98,298],{"x1":299,"y1":300,"x2":301,"y2":300,"stroke":169,"style":302},"136","65","186","stroke-width:2;marker-end:url(#bigjson-flow-arrow);",[83,304],{"x":301,"y":101,"width":305,"height":287,"rx":288,"fill":88,"stroke":116,"style":306},"180","stroke-width:2;",[90,308,311],{"x":309,"y":310,"fill":94,"style":295},"276","62","full body in RAM",[90,313,316],{"x":309,"y":314,"fill":169,"style":315},"79","text-anchor:middle;font-size:11;font-family:var(--font-sans);","420 MB of bytes",[98,318],{"x1":319,"y1":300,"x2":320,"y2":300,"stroke":169,"style":302},"366","416",[83,322],{"x":320,"y":101,"width":305,"height":287,"rx":288,"fill":88,"stroke":116,"style":306},[90,324,326],{"x":325,"y":310,"fill":94,"style":295},"506","1.2M live dicts",[90,328,329],{"x":325,"y":314,"fill":169,"style":315},"3.1 GB peak",[98,331],{"x1":332,"y1":300,"x2":333,"y2":300,"stroke":169,"style":302},"596","646",[90,335,337],{"x":336,"y":294,"fill":94,"style":295},"672","sink",[98,339],{"x1":92,"y1":340,"x2":341,"y2":340,"stroke":103,"style":104},"122","704",[90,343,345],{"x":92,"y":344,"fill":94,"style":95},"152","Streaming: memory is flat regardless of size",[83,347],{"x":92,"y":348,"width":286,"height":287,"rx":288,"fill":289,"stroke":103,"style":290},"168",[90,350,296],{"x":293,"y":351,"fill":94,"style":295},"196",[98,353],{"x1":299,"y1":354,"x2":355,"y2":354,"stroke":169,"style":302},"191","176",[83,357],{"x":355,"y":348,"width":358,"height":287,"rx":288,"fill":88,"stroke":132,"style":306},"130",[90,360,363],{"x":361,"y":362,"fill":94,"style":295},"241","188","64 KB chunk",[90,365,367],{"x":361,"y":366,"fill":169,"style":315},"205","reused buffer",[98,369],{"x1":370,"y1":354,"x2":371,"y2":354,"stroke":169,"style":302},"306","346",[83,373],{"x":371,"y":348,"width":374,"height":287,"rx":288,"fill":88,"stroke":146,"style":306},"140",[90,376,377],{"x":320,"y":362,"fill":94,"style":295},"event parser",[90,379,380],{"x":320,"y":366,"fill":169,"style":315},"yields one record",[98,382],{"x1":383,"y1":354,"x2":384,"y2":354,"stroke":169,"style":302},"486","526",[83,386],{"x":384,"y":348,"width":358,"height":287,"rx":288,"fill":88,"stroke":146,"style":306},[90,388,390],{"x":389,"y":362,"fill":94,"style":295},"591","batch of 5000",[90,392,393],{"x":389,"y":366,"fill":169,"style":315},"then flush",[278,395],{"d":396,"fill":397,"stroke":169,"style":398},"M591 214 L591 244 L241 244 L241 218","none","stroke-width:1.5;stroke-dasharray:4 3;marker-end:url(#bigjson-flow-arrow);",[90,400,402],{"x":320,"y":401,"fill":169,"style":315},"262","backpressure: the sink paces the socket read",[90,404,406],{"x":92,"y":405,"fill":169,"style":170},"286","Peak memory in the streaming lane is one chunk plus one batch, not one payload.",[14,408,409,410,412,413,416,417,419,420,423,424,428],{},"Here is the ",[18,411,33],{}," path against a live HTTP response. ",[18,414,415],{},"httpx"," gives you an async byte iterator; ",[18,418,33],{}," wants something with an ",[18,421,422],{},"async read",", so a nine-line adapter bridges them. Read ",[23,425,427],{"href":426},"\u002Fgetting-started-with-python-apis-for-builders\u002Fmaking-http-requests-with-requests-library\u002Fhttpx-vs-requests-for-async\u002F","httpx vs requests for async"," if you are still on a sync client — streaming imports are exactly the workload where the async client earns its keep.",[430,431,436],"pre",{"className":432,"code":433,"language":434,"meta":435,"style":435},"language-python shiki shiki-themes github-light github-dark","import os\nimport asyncio\nfrom typing import AsyncIterator\n\nimport httpx\nimport ijson\n\nEXPORT_URL = os.getenv(\"EXPORT_URL\", \"https:\u002F\u002Fexample.test\u002Fexports\u002Forders.json\")\nJSON_PATH = os.getenv(\"EXPORT_JSON_PATH\", \"orders.item\")\nBATCH_SIZE = int(os.getenv(\"IMPORT_BATCH_SIZE\", \"5000\"))\n\n\nclass AsyncByteReader:\n    \"\"\"Adapt an async byte iterator to the read() interface ijson expects.\"\"\"\n\n    def __init__(self, chunks: AsyncIterator[bytes]) -> None:\n        self._chunks = chunks.__aiter__()\n        self._buf = bytearray()\n\n    async def read(self, size: int = -1) -> bytes:\n        while size \u003C 0 or len(self._buf) \u003C size:\n            try:\n                self._buf += await self._chunks.__anext__()\n            except StopAsyncIteration:\n                break\n        if size \u003C 0:\n            size = len(self._buf)\n        out = bytes(self._buf[:size])\n        del self._buf[:size]\n        return out\n\n\nasync def stream_orders() -> None:\n    batch: list[dict] = []\n    timeout = httpx.Timeout(10.0, read=None)  # no read deadline on a long body\n    async with httpx.AsyncClient(timeout=timeout) as client:\n        async with client.stream(\"GET\", EXPORT_URL) as resp:\n            resp.raise_for_status()\n            reader = AsyncByteReader(resp.aiter_bytes(chunk_size=65536))\n            async for record in ijson.items_async(reader, JSON_PATH):\n                batch.append(record)\n                if len(batch) >= BATCH_SIZE:\n                    await flush(batch)\n                    batch.clear()\n    if batch:\n        await flush(batch)\n\n\nasync def flush(batch: list[dict]) -> None:\n    await asyncio.sleep(0)  # replace with your COPY \u002F bulk insert\n\n\nif __name__ == \"__main__\":\n    asyncio.run(stream_orders())\n","python","",[18,437,438,450,458,472,479,487,495,500,526,546,571,576,581,594,600,605,628,649,664,669,702,737,745,770,781,787,801,818,836,847,856,861,866,884,900,931,956,982,988,1009,1032,1038,1057,1066,1072,1081,1089,1094,1099,1120,1136,1141,1146,1163],{"__ignoreMap":435},[439,440,442,446],"span",{"class":98,"line":441},1,[439,443,445],{"class":444},"szBVR","import",[439,447,449],{"class":448},"sVt8B"," os\n",[439,451,453,455],{"class":98,"line":452},2,[439,454,445],{"class":444},[439,456,457],{"class":448}," asyncio\n",[439,459,461,464,467,469],{"class":98,"line":460},3,[439,462,463],{"class":444},"from",[439,465,466],{"class":448}," typing ",[439,468,445],{"class":444},[439,470,471],{"class":448}," AsyncIterator\n",[439,473,475],{"class":98,"line":474},4,[439,476,478],{"emptyLinePlaceholder":477},true,"\n",[439,480,482,484],{"class":98,"line":481},5,[439,483,445],{"class":444},[439,485,486],{"class":448}," httpx\n",[439,488,490,492],{"class":98,"line":489},6,[439,491,445],{"class":444},[439,493,494],{"class":448}," ijson\n",[439,496,498],{"class":98,"line":497},7,[439,499,478],{"emptyLinePlaceholder":477},[439,501,503,507,510,513,517,520,523],{"class":98,"line":502},8,[439,504,506],{"class":505},"sj4cs","EXPORT_URL",[439,508,509],{"class":444}," =",[439,511,512],{"class":448}," os.getenv(",[439,514,516],{"class":515},"sZZnC","\"EXPORT_URL\"",[439,518,519],{"class":448},", ",[439,521,522],{"class":515},"\"https:\u002F\u002Fexample.test\u002Fexports\u002Forders.json\"",[439,524,525],{"class":448},")\n",[439,527,529,532,534,536,539,541,544],{"class":98,"line":528},9,[439,530,531],{"class":505},"JSON_PATH",[439,533,509],{"class":444},[439,535,512],{"class":448},[439,537,538],{"class":515},"\"EXPORT_JSON_PATH\"",[439,540,519],{"class":448},[439,542,543],{"class":515},"\"orders.item\"",[439,545,525],{"class":448},[439,547,549,552,554,557,560,563,565,568],{"class":98,"line":548},10,[439,550,551],{"class":505},"BATCH_SIZE",[439,553,509],{"class":444},[439,555,556],{"class":505}," int",[439,558,559],{"class":448},"(os.getenv(",[439,561,562],{"class":515},"\"IMPORT_BATCH_SIZE\"",[439,564,519],{"class":448},[439,566,567],{"class":515},"\"5000\"",[439,569,570],{"class":448},"))\n",[439,572,574],{"class":98,"line":573},11,[439,575,478],{"emptyLinePlaceholder":477},[439,577,579],{"class":98,"line":578},12,[439,580,478],{"emptyLinePlaceholder":477},[439,582,584,587,591],{"class":98,"line":583},13,[439,585,586],{"class":444},"class",[439,588,590],{"class":589},"sScJk"," AsyncByteReader",[439,592,593],{"class":448},":\n",[439,595,597],{"class":98,"line":596},14,[439,598,599],{"class":515},"    \"\"\"Adapt an async byte iterator to the read() interface ijson expects.\"\"\"\n",[439,601,603],{"class":98,"line":602},15,[439,604,478],{"emptyLinePlaceholder":477},[439,606,608,611,614,617,620,623,626],{"class":98,"line":607},16,[439,609,610],{"class":444},"    def",[439,612,613],{"class":505}," __init__",[439,615,616],{"class":448},"(self, chunks: AsyncIterator[",[439,618,619],{"class":505},"bytes",[439,621,622],{"class":448},"]) -> ",[439,624,625],{"class":505},"None",[439,627,593],{"class":448},[439,629,631,634,637,640,643,646],{"class":98,"line":630},17,[439,632,633],{"class":505},"        self",[439,635,636],{"class":448},"._chunks ",[439,638,639],{"class":444},"=",[439,641,642],{"class":448}," chunks.",[439,644,645],{"class":505},"__aiter__",[439,647,648],{"class":448},"()\n",[439,650,652,654,657,659,662],{"class":98,"line":651},18,[439,653,633],{"class":505},[439,655,656],{"class":448},"._buf ",[439,658,639],{"class":444},[439,660,661],{"class":505}," bytearray",[439,663,648],{"class":448},[439,665,667],{"class":98,"line":666},19,[439,668,478],{"emptyLinePlaceholder":477},[439,670,672,675,678,681,684,687,689,692,695,698,700],{"class":98,"line":671},20,[439,673,674],{"class":444},"    async",[439,676,677],{"class":444}," def",[439,679,680],{"class":589}," read",[439,682,683],{"class":448},"(self, size: ",[439,685,686],{"class":505},"int",[439,688,509],{"class":444},[439,690,691],{"class":444}," -",[439,693,694],{"class":505},"1",[439,696,697],{"class":448},") -> ",[439,699,619],{"class":505},[439,701,593],{"class":448},[439,703,705,708,711,714,717,720,723,726,729,732,734],{"class":98,"line":704},21,[439,706,707],{"class":444},"        while",[439,709,710],{"class":448}," size ",[439,712,713],{"class":444},"\u003C",[439,715,716],{"class":505}," 0",[439,718,719],{"class":444}," or",[439,721,722],{"class":505}," len",[439,724,725],{"class":448},"(",[439,727,728],{"class":505},"self",[439,730,731],{"class":448},"._buf) ",[439,733,713],{"class":444},[439,735,736],{"class":448}," size:\n",[439,738,740,743],{"class":98,"line":739},22,[439,741,742],{"class":444},"            try",[439,744,593],{"class":448},[439,746,748,751,753,756,759,762,765,768],{"class":98,"line":747},23,[439,749,750],{"class":505},"                self",[439,752,656],{"class":448},[439,754,755],{"class":444},"+=",[439,757,758],{"class":444}," await",[439,760,761],{"class":505}," self",[439,763,764],{"class":448},"._chunks.",[439,766,767],{"class":505},"__anext__",[439,769,648],{"class":448},[439,771,773,776,779],{"class":98,"line":772},24,[439,774,775],{"class":444},"            except",[439,777,778],{"class":505}," StopAsyncIteration",[439,780,593],{"class":448},[439,782,784],{"class":98,"line":783},25,[439,785,786],{"class":444},"                break\n",[439,788,790,793,795,797,799],{"class":98,"line":789},26,[439,791,792],{"class":444},"        if",[439,794,710],{"class":448},[439,796,713],{"class":444},[439,798,716],{"class":505},[439,800,593],{"class":448},[439,802,804,807,809,811,813,815],{"class":98,"line":803},27,[439,805,806],{"class":448},"            size ",[439,808,639],{"class":444},[439,810,722],{"class":505},[439,812,725],{"class":448},[439,814,728],{"class":505},[439,816,817],{"class":448},"._buf)\n",[439,819,821,824,826,829,831,833],{"class":98,"line":820},28,[439,822,823],{"class":448},"        out ",[439,825,639],{"class":444},[439,827,828],{"class":505}," bytes",[439,830,725],{"class":448},[439,832,728],{"class":505},[439,834,835],{"class":448},"._buf[:size])\n",[439,837,839,842,844],{"class":98,"line":838},29,[439,840,841],{"class":444},"        del",[439,843,761],{"class":505},[439,845,846],{"class":448},"._buf[:size]\n",[439,848,850,853],{"class":98,"line":849},30,[439,851,852],{"class":444},"        return",[439,854,855],{"class":448}," out\n",[439,857,859],{"class":98,"line":858},31,[439,860,478],{"emptyLinePlaceholder":477},[439,862,864],{"class":98,"line":863},32,[439,865,478],{"emptyLinePlaceholder":477},[439,867,869,872,874,877,880,882],{"class":98,"line":868},33,[439,870,871],{"class":444},"async",[439,873,677],{"class":444},[439,875,876],{"class":589}," stream_orders",[439,878,879],{"class":448},"() -> ",[439,881,625],{"class":505},[439,883,593],{"class":448},[439,885,887,890,892,895,897],{"class":98,"line":886},34,[439,888,889],{"class":448},"    batch: list[",[439,891,54],{"class":505},[439,893,894],{"class":448},"] ",[439,896,639],{"class":444},[439,898,899],{"class":448}," []\n",[439,901,903,906,908,911,914,916,920,922,924,927],{"class":98,"line":902},35,[439,904,905],{"class":448},"    timeout ",[439,907,639],{"class":444},[439,909,910],{"class":448}," httpx.Timeout(",[439,912,913],{"class":505},"10.0",[439,915,519],{"class":448},[439,917,919],{"class":918},"s4XuR","read",[439,921,639],{"class":444},[439,923,625],{"class":505},[439,925,926],{"class":448},")  ",[439,928,930],{"class":929},"sJ8bj","# no read deadline on a long body\n",[439,932,934,936,939,942,945,947,950,953],{"class":98,"line":933},36,[439,935,674],{"class":444},[439,937,938],{"class":444}," with",[439,940,941],{"class":448}," httpx.AsyncClient(",[439,943,944],{"class":918},"timeout",[439,946,639],{"class":444},[439,948,949],{"class":448},"timeout) ",[439,951,952],{"class":444},"as",[439,954,955],{"class":448}," client:\n",[439,957,959,962,964,967,970,972,974,977,979],{"class":98,"line":958},37,[439,960,961],{"class":444},"        async",[439,963,938],{"class":444},[439,965,966],{"class":448}," client.stream(",[439,968,969],{"class":515},"\"GET\"",[439,971,519],{"class":448},[439,973,506],{"class":505},[439,975,976],{"class":448},") ",[439,978,952],{"class":444},[439,980,981],{"class":448}," resp:\n",[439,983,985],{"class":98,"line":984},38,[439,986,987],{"class":448},"            resp.raise_for_status()\n",[439,989,991,994,996,999,1002,1004,1007],{"class":98,"line":990},39,[439,992,993],{"class":448},"            reader ",[439,995,639],{"class":444},[439,997,998],{"class":448}," AsyncByteReader(resp.aiter_bytes(",[439,1000,1001],{"class":918},"chunk_size",[439,1003,639],{"class":444},[439,1005,1006],{"class":505},"65536",[439,1008,570],{"class":448},[439,1010,1012,1015,1018,1021,1024,1027,1029],{"class":98,"line":1011},40,[439,1013,1014],{"class":444},"            async",[439,1016,1017],{"class":444}," for",[439,1019,1020],{"class":448}," record ",[439,1022,1023],{"class":444},"in",[439,1025,1026],{"class":448}," ijson.items_async(reader, ",[439,1028,531],{"class":505},[439,1030,1031],{"class":448},"):\n",[439,1033,1035],{"class":98,"line":1034},41,[439,1036,1037],{"class":448},"                batch.append(record)\n",[439,1039,1041,1044,1046,1049,1052,1055],{"class":98,"line":1040},42,[439,1042,1043],{"class":444},"                if",[439,1045,722],{"class":505},[439,1047,1048],{"class":448},"(batch) ",[439,1050,1051],{"class":444},">=",[439,1053,1054],{"class":505}," BATCH_SIZE",[439,1056,593],{"class":448},[439,1058,1060,1063],{"class":98,"line":1059},43,[439,1061,1062],{"class":444},"                    await",[439,1064,1065],{"class":448}," flush(batch)\n",[439,1067,1069],{"class":98,"line":1068},44,[439,1070,1071],{"class":448},"                    batch.clear()\n",[439,1073,1075,1078],{"class":98,"line":1074},45,[439,1076,1077],{"class":444},"    if",[439,1079,1080],{"class":448}," batch:\n",[439,1082,1084,1087],{"class":98,"line":1083},46,[439,1085,1086],{"class":444},"        await",[439,1088,1065],{"class":448},[439,1090,1092],{"class":98,"line":1091},47,[439,1093,478],{"emptyLinePlaceholder":477},[439,1095,1097],{"class":98,"line":1096},48,[439,1098,478],{"emptyLinePlaceholder":477},[439,1100,1102,1104,1106,1109,1112,1114,1116,1118],{"class":98,"line":1101},49,[439,1103,871],{"class":444},[439,1105,677],{"class":444},[439,1107,1108],{"class":589}," flush",[439,1110,1111],{"class":448},"(batch: list[",[439,1113,54],{"class":505},[439,1115,622],{"class":448},[439,1117,625],{"class":505},[439,1119,593],{"class":448},[439,1121,1123,1126,1129,1131,1133],{"class":98,"line":1122},50,[439,1124,1125],{"class":444},"    await",[439,1127,1128],{"class":448}," asyncio.sleep(",[439,1130,85],{"class":505},[439,1132,926],{"class":448},[439,1134,1135],{"class":929},"# replace with your COPY \u002F bulk insert\n",[439,1137,1139],{"class":98,"line":1138},51,[439,1140,478],{"emptyLinePlaceholder":477},[439,1142,1144],{"class":98,"line":1143},52,[439,1145,478],{"emptyLinePlaceholder":477},[439,1147,1149,1152,1155,1158,1161],{"class":98,"line":1148},53,[439,1150,1151],{"class":444},"if",[439,1153,1154],{"class":505}," __name__",[439,1156,1157],{"class":444}," ==",[439,1159,1160],{"class":515}," \"__main__\"",[439,1162,593],{"class":448},[439,1164,1166],{"class":98,"line":1165},54,[439,1167,1168],{"class":448},"    asyncio.run(stream_orders())\n",[14,1170,1171,1172,1176],{},"The NDJSON path is shorter because the format does the work. Each line is a complete document, so you never hold more than one record plus the current batch, and validation slots straight in — pair it with ",[23,1173,1175],{"href":1174},"\u002Fgetting-started-with-python-apis-for-builders\u002Fparsing-json-responses\u002Fvalidating-json-with-pydantic-v2\u002F","Pydantic v2 validation"," so a malformed line fails at the boundary instead of corrupting a bulk insert.",[430,1178,1180],{"className":432,"code":1179,"language":434,"meta":435,"style":435},"import os\nimport asyncio\n\nimport httpx\nfrom pydantic import BaseModel, ValidationError\n\nNDJSON_URL = os.getenv(\"NDJSON_URL\", \"https:\u002F\u002Fexample.test\u002Fexports\u002Forders.ndjson\")\nMAX_BAD_LINES = int(os.getenv(\"IMPORT_MAX_BAD_LINES\", \"50\"))\n\n\nclass Order(BaseModel):\n    id: int\n    email: str\n    total_cents: int\n\n\nasync def import_ndjson() -> tuple[int, int]:\n    ok = bad = 0\n    async with httpx.AsyncClient(timeout=httpx.Timeout(10.0, read=None)) as client:\n        async with client.stream(\"GET\", NDJSON_URL) as resp:\n            resp.raise_for_status()\n            async for line in resp.aiter_lines():\n                if not line.strip():\n                    continue\n                try:\n                    Order.model_validate_json(line)\n                except ValidationError:\n                    bad += 1\n                    if bad > MAX_BAD_LINES:\n                        raise RuntimeError(\"upstream export is malformed\")\n                    continue\n                ok += 1\n    return ok, bad\n\n\nif __name__ == \"__main__\":\n    print(asyncio.run(import_ndjson()))\n",[18,1181,1182,1188,1194,1198,1204,1216,1220,1239,1260,1264,1268,1282,1293,1301,1308,1312,1316,1337,1352,1384,1404,1408,1422,1432,1437,1444,1449,1457,1467,1482,1497,1501,1510,1518,1522,1526,1538],{"__ignoreMap":435},[439,1183,1184,1186],{"class":98,"line":441},[439,1185,445],{"class":444},[439,1187,449],{"class":448},[439,1189,1190,1192],{"class":98,"line":452},[439,1191,445],{"class":444},[439,1193,457],{"class":448},[439,1195,1196],{"class":98,"line":460},[439,1197,478],{"emptyLinePlaceholder":477},[439,1199,1200,1202],{"class":98,"line":474},[439,1201,445],{"class":444},[439,1203,486],{"class":448},[439,1205,1206,1208,1211,1213],{"class":98,"line":481},[439,1207,463],{"class":444},[439,1209,1210],{"class":448}," pydantic ",[439,1212,445],{"class":444},[439,1214,1215],{"class":448}," BaseModel, ValidationError\n",[439,1217,1218],{"class":98,"line":489},[439,1219,478],{"emptyLinePlaceholder":477},[439,1221,1222,1225,1227,1229,1232,1234,1237],{"class":98,"line":497},[439,1223,1224],{"class":505},"NDJSON_URL",[439,1226,509],{"class":444},[439,1228,512],{"class":448},[439,1230,1231],{"class":515},"\"NDJSON_URL\"",[439,1233,519],{"class":448},[439,1235,1236],{"class":515},"\"https:\u002F\u002Fexample.test\u002Fexports\u002Forders.ndjson\"",[439,1238,525],{"class":448},[439,1240,1241,1244,1246,1248,1250,1253,1255,1258],{"class":98,"line":502},[439,1242,1243],{"class":505},"MAX_BAD_LINES",[439,1245,509],{"class":444},[439,1247,556],{"class":505},[439,1249,559],{"class":448},[439,1251,1252],{"class":515},"\"IMPORT_MAX_BAD_LINES\"",[439,1254,519],{"class":448},[439,1256,1257],{"class":515},"\"50\"",[439,1259,570],{"class":448},[439,1261,1262],{"class":98,"line":528},[439,1263,478],{"emptyLinePlaceholder":477},[439,1265,1266],{"class":98,"line":548},[439,1267,478],{"emptyLinePlaceholder":477},[439,1269,1270,1272,1275,1277,1280],{"class":98,"line":573},[439,1271,586],{"class":444},[439,1273,1274],{"class":589}," Order",[439,1276,725],{"class":448},[439,1278,1279],{"class":589},"BaseModel",[439,1281,1031],{"class":448},[439,1283,1284,1287,1290],{"class":98,"line":578},[439,1285,1286],{"class":505},"    id",[439,1288,1289],{"class":448},": ",[439,1291,1292],{"class":505},"int\n",[439,1294,1295,1298],{"class":98,"line":583},[439,1296,1297],{"class":448},"    email: ",[439,1299,1300],{"class":505},"str\n",[439,1302,1303,1306],{"class":98,"line":596},[439,1304,1305],{"class":448},"    total_cents: ",[439,1307,1292],{"class":505},[439,1309,1310],{"class":98,"line":602},[439,1311,478],{"emptyLinePlaceholder":477},[439,1313,1314],{"class":98,"line":607},[439,1315,478],{"emptyLinePlaceholder":477},[439,1317,1318,1320,1322,1325,1328,1330,1332,1334],{"class":98,"line":630},[439,1319,871],{"class":444},[439,1321,677],{"class":444},[439,1323,1324],{"class":589}," import_ndjson",[439,1326,1327],{"class":448},"() -> tuple[",[439,1329,686],{"class":505},[439,1331,519],{"class":448},[439,1333,686],{"class":505},[439,1335,1336],{"class":448},"]:\n",[439,1338,1339,1342,1344,1347,1349],{"class":98,"line":651},[439,1340,1341],{"class":448},"    ok ",[439,1343,639],{"class":444},[439,1345,1346],{"class":448}," bad ",[439,1348,639],{"class":444},[439,1350,1351],{"class":505}," 0\n",[439,1353,1354,1356,1358,1360,1362,1364,1367,1369,1371,1373,1375,1377,1380,1382],{"class":98,"line":666},[439,1355,674],{"class":444},[439,1357,938],{"class":444},[439,1359,941],{"class":448},[439,1361,944],{"class":918},[439,1363,639],{"class":444},[439,1365,1366],{"class":448},"httpx.Timeout(",[439,1368,913],{"class":505},[439,1370,519],{"class":448},[439,1372,919],{"class":918},[439,1374,639],{"class":444},[439,1376,625],{"class":505},[439,1378,1379],{"class":448},")) ",[439,1381,952],{"class":444},[439,1383,955],{"class":448},[439,1385,1386,1388,1390,1392,1394,1396,1398,1400,1402],{"class":98,"line":671},[439,1387,961],{"class":444},[439,1389,938],{"class":444},[439,1391,966],{"class":448},[439,1393,969],{"class":515},[439,1395,519],{"class":448},[439,1397,1224],{"class":505},[439,1399,976],{"class":448},[439,1401,952],{"class":444},[439,1403,981],{"class":448},[439,1405,1406],{"class":98,"line":704},[439,1407,987],{"class":448},[439,1409,1410,1412,1414,1417,1419],{"class":98,"line":739},[439,1411,1014],{"class":444},[439,1413,1017],{"class":444},[439,1415,1416],{"class":448}," line ",[439,1418,1023],{"class":444},[439,1420,1421],{"class":448}," resp.aiter_lines():\n",[439,1423,1424,1426,1429],{"class":98,"line":747},[439,1425,1043],{"class":444},[439,1427,1428],{"class":444}," not",[439,1430,1431],{"class":448}," line.strip():\n",[439,1433,1434],{"class":98,"line":772},[439,1435,1436],{"class":444},"                    continue\n",[439,1438,1439,1442],{"class":98,"line":783},[439,1440,1441],{"class":444},"                try",[439,1443,593],{"class":448},[439,1445,1446],{"class":98,"line":789},[439,1447,1448],{"class":448},"                    Order.model_validate_json(line)\n",[439,1450,1451,1454],{"class":98,"line":803},[439,1452,1453],{"class":444},"                except",[439,1455,1456],{"class":448}," ValidationError:\n",[439,1458,1459,1462,1464],{"class":98,"line":820},[439,1460,1461],{"class":448},"                    bad ",[439,1463,755],{"class":444},[439,1465,1466],{"class":505}," 1\n",[439,1468,1469,1472,1474,1477,1480],{"class":98,"line":838},[439,1470,1471],{"class":444},"                    if",[439,1473,1346],{"class":448},[439,1475,1476],{"class":444},">",[439,1478,1479],{"class":505}," MAX_BAD_LINES",[439,1481,593],{"class":448},[439,1483,1484,1487,1490,1492,1495],{"class":98,"line":849},[439,1485,1486],{"class":444},"                        raise",[439,1488,1489],{"class":505}," RuntimeError",[439,1491,725],{"class":448},[439,1493,1494],{"class":515},"\"upstream export is malformed\"",[439,1496,525],{"class":448},[439,1498,1499],{"class":98,"line":858},[439,1500,1436],{"class":444},[439,1502,1503,1506,1508],{"class":98,"line":863},[439,1504,1505],{"class":448},"                ok ",[439,1507,755],{"class":444},[439,1509,1466],{"class":505},[439,1511,1512,1515],{"class":98,"line":868},[439,1513,1514],{"class":444},"    return",[439,1516,1517],{"class":448}," ok, bad\n",[439,1519,1520],{"class":98,"line":886},[439,1521,478],{"emptyLinePlaceholder":477},[439,1523,1524],{"class":98,"line":902},[439,1525,478],{"emptyLinePlaceholder":477},[439,1527,1528,1530,1532,1534,1536],{"class":98,"line":933},[439,1529,1151],{"class":444},[439,1531,1154],{"class":505},[439,1533,1157],{"class":444},[439,1535,1160],{"class":515},[439,1537,593],{"class":448},[439,1539,1540,1543],{"class":98,"line":958},[439,1541,1542],{"class":505},"    print",[439,1544,1545],{"class":448},"(asyncio.run(import_ndjson()))\n",[14,1547,1548,1549,1552],{},"That ",[18,1550,1551],{},"bad"," counter matters more than it looks. A whole-document parse is all-or-nothing: one stray byte at offset 400,000,000 discards the entire job. NDJSON degrades gracefully — skip the bad line, log it, keep the other 1.2 million records.",[40,1554],{},[43,1556,1558],{"id":1557},"measure-before-you-migrate","Measure Before You Migrate",[14,1560,1561],{},"Do not switch on vibes. Two numbers justify the work to yourself and to whoever pays the hosting bill: peak resident memory and time to first record. The second one is the sleeper. A buffered parse produces nothing for 11.4 seconds; a streaming parse hands you a usable record in 0.3 seconds, which is the difference between a request that can report progress and one that looks hung.",[430,1563,1565],{"className":432,"code":1564,"language":434,"meta":435,"style":435},"import os\nimport resource\nimport time\nimport tracemalloc\nfrom collections.abc import Callable, Iterator\n\nSAMPLE_PATH = os.getenv(\"SAMPLE_JSON_PATH\", \".\u002Fsample.json\")\n\n\ndef profile(name: str, run: Callable[[], Iterator[dict]]) -> None:\n    tracemalloc.start()\n    start = time.perf_counter()\n    first_at: float | None = None\n    count = 0\n    for _ in run():\n        if first_at is None:\n            first_at = time.perf_counter() - start\n        count += 1\n    wall = time.perf_counter() - start\n    _, py_peak = tracemalloc.get_traced_memory()\n    tracemalloc.stop()\n    rss_mb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss \u002F 1024\n    print(\n        f\"{name:>16} | records={count} | first={first_at:.2f}s \"\n        f\"| wall={wall:.2f}s | py_peak={py_peak \u002F 1_048_576:.0f}MB \"\n        f\"| max_rss={rss_mb:.0f}MB\"\n    )\n",[18,1566,1567,1573,1580,1587,1594,1606,1610,1629,1633,1637,1662,1667,1677,1696,1705,1718,1732,1748,1757,1770,1780,1785,1807,1814,1860,1897,1916],{"__ignoreMap":435},[439,1568,1569,1571],{"class":98,"line":441},[439,1570,445],{"class":444},[439,1572,449],{"class":448},[439,1574,1575,1577],{"class":98,"line":452},[439,1576,445],{"class":444},[439,1578,1579],{"class":448}," resource\n",[439,1581,1582,1584],{"class":98,"line":460},[439,1583,445],{"class":444},[439,1585,1586],{"class":448}," time\n",[439,1588,1589,1591],{"class":98,"line":474},[439,1590,445],{"class":444},[439,1592,1593],{"class":448}," tracemalloc\n",[439,1595,1596,1598,1601,1603],{"class":98,"line":481},[439,1597,463],{"class":444},[439,1599,1600],{"class":448}," collections.abc ",[439,1602,445],{"class":444},[439,1604,1605],{"class":448}," Callable, Iterator\n",[439,1607,1608],{"class":98,"line":489},[439,1609,478],{"emptyLinePlaceholder":477},[439,1611,1612,1615,1617,1619,1622,1624,1627],{"class":98,"line":497},[439,1613,1614],{"class":505},"SAMPLE_PATH",[439,1616,509],{"class":444},[439,1618,512],{"class":448},[439,1620,1621],{"class":515},"\"SAMPLE_JSON_PATH\"",[439,1623,519],{"class":448},[439,1625,1626],{"class":515},"\".\u002Fsample.json\"",[439,1628,525],{"class":448},[439,1630,1631],{"class":98,"line":502},[439,1632,478],{"emptyLinePlaceholder":477},[439,1634,1635],{"class":98,"line":528},[439,1636,478],{"emptyLinePlaceholder":477},[439,1638,1639,1642,1645,1648,1650,1653,1655,1658,1660],{"class":98,"line":548},[439,1640,1641],{"class":444},"def",[439,1643,1644],{"class":589}," profile",[439,1646,1647],{"class":448},"(name: ",[439,1649,58],{"class":505},[439,1651,1652],{"class":448},", run: Callable[[], Iterator[",[439,1654,54],{"class":505},[439,1656,1657],{"class":448},"]]) -> ",[439,1659,625],{"class":505},[439,1661,593],{"class":448},[439,1663,1664],{"class":98,"line":573},[439,1665,1666],{"class":448},"    tracemalloc.start()\n",[439,1668,1669,1672,1674],{"class":98,"line":578},[439,1670,1671],{"class":448},"    start ",[439,1673,639],{"class":444},[439,1675,1676],{"class":448}," time.perf_counter()\n",[439,1678,1679,1682,1685,1688,1691,1693],{"class":98,"line":583},[439,1680,1681],{"class":448},"    first_at: ",[439,1683,1684],{"class":505},"float",[439,1686,1687],{"class":444}," |",[439,1689,1690],{"class":505}," None",[439,1692,509],{"class":444},[439,1694,1695],{"class":505}," None\n",[439,1697,1698,1701,1703],{"class":98,"line":596},[439,1699,1700],{"class":448},"    count ",[439,1702,639],{"class":444},[439,1704,1351],{"class":505},[439,1706,1707,1710,1713,1715],{"class":98,"line":602},[439,1708,1709],{"class":444},"    for",[439,1711,1712],{"class":448}," _ ",[439,1714,1023],{"class":444},[439,1716,1717],{"class":448}," run():\n",[439,1719,1720,1722,1725,1728,1730],{"class":98,"line":607},[439,1721,792],{"class":444},[439,1723,1724],{"class":448}," first_at ",[439,1726,1727],{"class":444},"is",[439,1729,1690],{"class":505},[439,1731,593],{"class":448},[439,1733,1734,1737,1739,1742,1745],{"class":98,"line":630},[439,1735,1736],{"class":448},"            first_at ",[439,1738,639],{"class":444},[439,1740,1741],{"class":448}," time.perf_counter() ",[439,1743,1744],{"class":444},"-",[439,1746,1747],{"class":448}," start\n",[439,1749,1750,1753,1755],{"class":98,"line":651},[439,1751,1752],{"class":448},"        count ",[439,1754,755],{"class":444},[439,1756,1466],{"class":505},[439,1758,1759,1762,1764,1766,1768],{"class":98,"line":666},[439,1760,1761],{"class":448},"    wall ",[439,1763,639],{"class":444},[439,1765,1741],{"class":448},[439,1767,1744],{"class":444},[439,1769,1747],{"class":448},[439,1771,1772,1775,1777],{"class":98,"line":671},[439,1773,1774],{"class":448},"    _, py_peak ",[439,1776,639],{"class":444},[439,1778,1779],{"class":448}," tracemalloc.get_traced_memory()\n",[439,1781,1782],{"class":98,"line":704},[439,1783,1784],{"class":448},"    tracemalloc.stop()\n",[439,1786,1787,1790,1792,1795,1798,1801,1804],{"class":98,"line":739},[439,1788,1789],{"class":448},"    rss_mb ",[439,1791,639],{"class":444},[439,1793,1794],{"class":448}," resource.getrusage(resource.",[439,1796,1797],{"class":505},"RUSAGE_SELF",[439,1799,1800],{"class":448},").ru_maxrss ",[439,1802,1803],{"class":444},"\u002F",[439,1805,1806],{"class":505}," 1024\n",[439,1808,1809,1811],{"class":98,"line":747},[439,1810,1542],{"class":505},[439,1812,1813],{"class":448},"(\n",[439,1815,1816,1819,1822,1825,1828,1831,1834,1837,1839,1842,1844,1847,1849,1852,1855,1857],{"class":98,"line":772},[439,1817,1818],{"class":444},"        f",[439,1820,1821],{"class":515},"\"",[439,1823,1824],{"class":505},"{",[439,1826,1827],{"class":448},"name",[439,1829,1830],{"class":444},":>16",[439,1832,1833],{"class":505},"}",[439,1835,1836],{"class":515}," | records=",[439,1838,1824],{"class":505},[439,1840,1841],{"class":448},"count",[439,1843,1833],{"class":505},[439,1845,1846],{"class":515}," | first=",[439,1848,1824],{"class":505},[439,1850,1851],{"class":448},"first_at",[439,1853,1854],{"class":444},":.2f",[439,1856,1833],{"class":505},[439,1858,1859],{"class":515},"s \"\n",[439,1861,1862,1864,1867,1869,1872,1874,1876,1879,1881,1884,1886,1889,1892,1894],{"class":98,"line":783},[439,1863,1818],{"class":444},[439,1865,1866],{"class":515},"\"| wall=",[439,1868,1824],{"class":505},[439,1870,1871],{"class":448},"wall",[439,1873,1854],{"class":444},[439,1875,1833],{"class":505},[439,1877,1878],{"class":515},"s | py_peak=",[439,1880,1824],{"class":505},[439,1882,1883],{"class":448},"py_peak ",[439,1885,1803],{"class":444},[439,1887,1888],{"class":505}," 1_048_576",[439,1890,1891],{"class":444},":.0f",[439,1893,1833],{"class":505},[439,1895,1896],{"class":515},"MB \"\n",[439,1898,1899,1901,1904,1906,1909,1911,1913],{"class":98,"line":789},[439,1900,1818],{"class":444},[439,1902,1903],{"class":515},"\"| max_rss=",[439,1905,1824],{"class":505},[439,1907,1908],{"class":448},"rss_mb",[439,1910,1891],{"class":444},[439,1912,1833],{"class":505},[439,1914,1915],{"class":515},"MB\"\n",[439,1917,1918],{"class":98,"line":803},[439,1919,1920],{"class":448},"    )\n",[14,1922,1923,1924,1926,1927,1929,1930,1934,1935,1937,1938,1941,1942,1945],{},"Run each candidate through that harness against the same file. On the 420 MB fixture the results were unambiguous: ",[18,1925,37],{}," finished the fastest overall at 11.4 s wall but needed 3.1 GB; ",[18,1928,33],{}," held 48 MB and took 26.8 s; the NDJSON loop held 31 MB and finished in 9.2 s — faster ",[1931,1932,1933],"em",{},"and"," leaner than the buffered parse, because it never builds the intermediate whole-document structure. Swap ",[18,1936,37],{}," for ",[18,1939,1940],{},"orjson.loads"," inside the NDJSON loop and it drops to about 6.1 s at the same memory. Note that ",[18,1943,1944],{},"ru_maxrss"," is a high-water mark for the whole process, so run each strategy in its own process or the first bad number poisons the rest.",[64,1947,73,1952,73,1955,73,1958,73,1961,73,1964,73,1967,73,1972,73,1977,73,1982,73,1986,73,1990,73,1994,73,1997,73,2001,73,2006,73,2010,73,2014,73,2017,73,2020,73,2023,73,2027,73,2031,73,2035,73,2039],{"viewBox":1948,"role":67,"ariaLabelledBy":1949,"xmlns":71,"style":72},"0 0 720 240",[1950,1951],"bigjson-ttfr-t","bigjson-ttfr-d",[75,1953,1954],{"id":1950},"Time to first record, buffered versus streaming",[79,1956,1957],{"id":1951},"Timeline showing the buffered parse downloading for 6.5 seconds then parsing until 11.4 seconds before the first record, while the streaming parse emits its first record at 0.3 seconds and works continuously to 26.8 seconds.",[83,1959],{"x":85,"y":85,"width":86,"height":1960,"fill":88},"240",[90,1962,1963],{"x":92,"y":93,"fill":94,"style":95},"Wall clock on a 420 MB export (seconds)",[90,1965,37],{"x":92,"y":1966,"fill":94,"style":121},"66",[83,1968],{"x":1969,"y":293,"width":1970,"height":1971,"rx":115,"fill":289,"stroke":103,"style":290},"60","134","24",[90,1973,1976],{"x":1974,"y":1975,"fill":94,"style":315},"127","93","download",[83,1978],{"x":1979,"y":293,"width":1980,"height":1971,"rx":115,"fill":116,"fillOpacity":1981,"stroke":116,"style":290},"194","101",0.22,[90,1983,1985],{"x":1984,"y":1975,"fill":94,"style":315},"244","parse",[98,1987],{"x1":1988,"y1":294,"x2":1988,"y2":1989,"stroke":94,"style":306},"295","112",[90,1991,1993],{"x":1992,"y":1989,"fill":94,"style":170},"303","first record at 11.4s",[90,1995,33],{"x":92,"y":1996,"fill":94,"style":121},"146",[83,1998],{"x":1969,"y":1999,"width":2000,"height":1971,"rx":115,"fill":146,"fillOpacity":1981,"stroke":146,"style":290},"156","554",[90,2002,2005],{"x":2003,"y":2004,"fill":94,"style":315},"337","173","download and emit records continuously",[98,2007],{"x1":1966,"y1":2008,"x2":1966,"y2":2009,"stroke":94,"style":306},"150","192",[90,2011,2013],{"x":120,"y":2012,"fill":94,"style":170},"204","first record at 0.3s",[98,2015],{"x1":1969,"y1":2016,"x2":165,"y2":2016,"stroke":103,"style":104},"214",[90,2018,85],{"x":1969,"y":2019,"fill":169,"style":315},"231",[90,2021,145],{"x":2022,"y":2019,"fill":169,"style":315},"163",[90,2024,2026],{"x":2025,"y":2019,"fill":169,"style":315},"267","10",[90,2028,2030],{"x":2029,"y":2019,"fill":169,"style":315},"370","15",[90,2032,2034],{"x":2033,"y":2019,"fill":169,"style":315},"473","20",[90,2036,2038],{"x":2037,"y":2019,"fill":169,"style":315},"577","25",[90,2040,2042],{"x":2041,"y":2019,"fill":169,"style":315},"680","30",[40,2044],{},[43,2046,2048],{"id":2047},"accepting-large-payloads-on-your-own-api","Accepting Large Payloads on Your Own API",[14,2050,2051,2052,2056,2057,2060],{},"Reading big JSON is half the job; the other half is letting customers send it to you. The default in ",[23,2053,2055],{"href":2054},"\u002Fgetting-started-with-python-apis-for-builders\u002Fsetting-up-fastapi\u002F","FastAPI"," — a typed body model on the handler signature — buffers the entire request before your function runs, so a 400 MB upload kills the worker before a single line of your code executes. Drop to ",[18,2058,2059],{},"Request.stream()"," and consume the body as it arrives.",[430,2062,2064],{"className":432,"code":2063,"language":434,"meta":435,"style":435},"import os\n\nfrom fastapi import FastAPI, HTTPException, Request\nfrom pydantic import BaseModel, ValidationError\n\napp = FastAPI()\nMAX_BYTES = int(os.getenv(\"IMPORT_MAX_BYTES\", \"2_000_000_000\"))\nBATCH_SIZE = int(os.getenv(\"IMPORT_BATCH_SIZE\", \"5000\"))\n\n\nclass Order(BaseModel):\n    id: int\n    email: str\n    total_cents: int\n\n\n@app.post(\"\u002Fimports\u002Forders\", status_code=202)\nasync def import_orders(request: Request) -> dict[str, int]:\n    buf = bytearray()\n    batch: list[Order] = []\n    total = accepted = 0\n\n    async for chunk in request.stream():\n        total += len(chunk)\n        if total > MAX_BYTES:\n            raise HTTPException(status_code=413, detail=\"payload too large\")\n        buf += chunk\n        while (nl := buf.find(b\"\\n\")) != -1:\n            line = bytes(buf[:nl])\n            del buf[: nl + 1]\n            if not line.strip():\n                continue\n            try:\n                batch.append(Order.model_validate_json(line))\n            except ValidationError as exc:\n                raise HTTPException(status_code=422, detail=exc.errors()[:5])\n            accepted += 1\n            if len(batch) >= BATCH_SIZE:\n                await persist(batch)\n                batch.clear()\n\n    if buf.strip():\n        batch.append(Order.model_validate_json(bytes(buf)))\n        accepted += 1\n    if batch:\n        await persist(batch)\n    return {\"accepted\": accepted}\n\n\nasync def persist(batch: list[Order]) -> None:\n    ...  # bulk insert; see the async SQLAlchemy guide\n",[18,2065,2066,2072,2076,2088,2098,2102,2112,2133,2151,2155,2159,2171,2179,2185,2191,2195,2199,2221,2241,2252,2261,2275,2279,2293,2305,2319,2346,2356,2390,2402,2419,2428,2433,2439,2444,2456,2484,2493,2507,2515,2520,2524,2531,2541,2550,2556,2562,2575,2579,2583,2599],{"__ignoreMap":435},[439,2067,2068,2070],{"class":98,"line":441},[439,2069,445],{"class":444},[439,2071,449],{"class":448},[439,2073,2074],{"class":98,"line":452},[439,2075,478],{"emptyLinePlaceholder":477},[439,2077,2078,2080,2083,2085],{"class":98,"line":460},[439,2079,463],{"class":444},[439,2081,2082],{"class":448}," fastapi ",[439,2084,445],{"class":444},[439,2086,2087],{"class":448}," FastAPI, HTTPException, Request\n",[439,2089,2090,2092,2094,2096],{"class":98,"line":474},[439,2091,463],{"class":444},[439,2093,1210],{"class":448},[439,2095,445],{"class":444},[439,2097,1215],{"class":448},[439,2099,2100],{"class":98,"line":481},[439,2101,478],{"emptyLinePlaceholder":477},[439,2103,2104,2107,2109],{"class":98,"line":489},[439,2105,2106],{"class":448},"app ",[439,2108,639],{"class":444},[439,2110,2111],{"class":448}," FastAPI()\n",[439,2113,2114,2117,2119,2121,2123,2126,2128,2131],{"class":98,"line":497},[439,2115,2116],{"class":505},"MAX_BYTES",[439,2118,509],{"class":444},[439,2120,556],{"class":505},[439,2122,559],{"class":448},[439,2124,2125],{"class":515},"\"IMPORT_MAX_BYTES\"",[439,2127,519],{"class":448},[439,2129,2130],{"class":515},"\"2_000_000_000\"",[439,2132,570],{"class":448},[439,2134,2135,2137,2139,2141,2143,2145,2147,2149],{"class":98,"line":502},[439,2136,551],{"class":505},[439,2138,509],{"class":444},[439,2140,556],{"class":505},[439,2142,559],{"class":448},[439,2144,562],{"class":515},[439,2146,519],{"class":448},[439,2148,567],{"class":515},[439,2150,570],{"class":448},[439,2152,2153],{"class":98,"line":528},[439,2154,478],{"emptyLinePlaceholder":477},[439,2156,2157],{"class":98,"line":548},[439,2158,478],{"emptyLinePlaceholder":477},[439,2160,2161,2163,2165,2167,2169],{"class":98,"line":573},[439,2162,586],{"class":444},[439,2164,1274],{"class":589},[439,2166,725],{"class":448},[439,2168,1279],{"class":589},[439,2170,1031],{"class":448},[439,2172,2173,2175,2177],{"class":98,"line":578},[439,2174,1286],{"class":505},[439,2176,1289],{"class":448},[439,2178,1292],{"class":505},[439,2180,2181,2183],{"class":98,"line":583},[439,2182,1297],{"class":448},[439,2184,1300],{"class":505},[439,2186,2187,2189],{"class":98,"line":596},[439,2188,1305],{"class":448},[439,2190,1292],{"class":505},[439,2192,2193],{"class":98,"line":602},[439,2194,478],{"emptyLinePlaceholder":477},[439,2196,2197],{"class":98,"line":607},[439,2198,478],{"emptyLinePlaceholder":477},[439,2200,2201,2204,2206,2209,2211,2214,2216,2219],{"class":98,"line":630},[439,2202,2203],{"class":589},"@app.post",[439,2205,725],{"class":448},[439,2207,2208],{"class":515},"\"\u002Fimports\u002Forders\"",[439,2210,519],{"class":448},[439,2212,2213],{"class":918},"status_code",[439,2215,639],{"class":444},[439,2217,2218],{"class":505},"202",[439,2220,525],{"class":448},[439,2222,2223,2225,2227,2230,2233,2235,2237,2239],{"class":98,"line":651},[439,2224,871],{"class":444},[439,2226,677],{"class":444},[439,2228,2229],{"class":589}," import_orders",[439,2231,2232],{"class":448},"(request: Request) -> dict[",[439,2234,58],{"class":505},[439,2236,519],{"class":448},[439,2238,686],{"class":505},[439,2240,1336],{"class":448},[439,2242,2243,2246,2248,2250],{"class":98,"line":666},[439,2244,2245],{"class":448},"    buf ",[439,2247,639],{"class":444},[439,2249,661],{"class":505},[439,2251,648],{"class":448},[439,2253,2254,2257,2259],{"class":98,"line":671},[439,2255,2256],{"class":448},"    batch: list[Order] ",[439,2258,639],{"class":444},[439,2260,899],{"class":448},[439,2262,2263,2266,2268,2271,2273],{"class":98,"line":704},[439,2264,2265],{"class":448},"    total ",[439,2267,639],{"class":444},[439,2269,2270],{"class":448}," accepted ",[439,2272,639],{"class":444},[439,2274,1351],{"class":505},[439,2276,2277],{"class":98,"line":739},[439,2278,478],{"emptyLinePlaceholder":477},[439,2280,2281,2283,2285,2288,2290],{"class":98,"line":747},[439,2282,674],{"class":444},[439,2284,1017],{"class":444},[439,2286,2287],{"class":448}," chunk ",[439,2289,1023],{"class":444},[439,2291,2292],{"class":448}," request.stream():\n",[439,2294,2295,2298,2300,2302],{"class":98,"line":772},[439,2296,2297],{"class":448},"        total ",[439,2299,755],{"class":444},[439,2301,722],{"class":505},[439,2303,2304],{"class":448},"(chunk)\n",[439,2306,2307,2309,2312,2314,2317],{"class":98,"line":783},[439,2308,792],{"class":444},[439,2310,2311],{"class":448}," total ",[439,2313,1476],{"class":444},[439,2315,2316],{"class":505}," MAX_BYTES",[439,2318,593],{"class":448},[439,2320,2321,2324,2327,2329,2331,2334,2336,2339,2341,2344],{"class":98,"line":789},[439,2322,2323],{"class":444},"            raise",[439,2325,2326],{"class":448}," HTTPException(",[439,2328,2213],{"class":918},[439,2330,639],{"class":444},[439,2332,2333],{"class":505},"413",[439,2335,519],{"class":448},[439,2337,2338],{"class":918},"detail",[439,2340,639],{"class":444},[439,2342,2343],{"class":515},"\"payload too large\"",[439,2345,525],{"class":448},[439,2347,2348,2351,2353],{"class":98,"line":803},[439,2349,2350],{"class":448},"        buf ",[439,2352,755],{"class":444},[439,2354,2355],{"class":448}," chunk\n",[439,2357,2358,2360,2363,2366,2369,2372,2374,2377,2379,2381,2384,2386,2388],{"class":98,"line":820},[439,2359,707],{"class":444},[439,2361,2362],{"class":448}," (nl ",[439,2364,2365],{"class":444},":=",[439,2367,2368],{"class":448}," buf.find(",[439,2370,2371],{"class":444},"b",[439,2373,1821],{"class":515},[439,2375,2376],{"class":505},"\\n",[439,2378,1821],{"class":515},[439,2380,1379],{"class":448},[439,2382,2383],{"class":444},"!=",[439,2385,691],{"class":444},[439,2387,694],{"class":505},[439,2389,593],{"class":448},[439,2391,2392,2395,2397,2399],{"class":98,"line":838},[439,2393,2394],{"class":448},"            line ",[439,2396,639],{"class":444},[439,2398,828],{"class":505},[439,2400,2401],{"class":448},"(buf[:nl])\n",[439,2403,2404,2407,2410,2413,2416],{"class":98,"line":849},[439,2405,2406],{"class":444},"            del",[439,2408,2409],{"class":448}," buf[: nl ",[439,2411,2412],{"class":444},"+",[439,2414,2415],{"class":505}," 1",[439,2417,2418],{"class":448},"]\n",[439,2420,2421,2424,2426],{"class":98,"line":858},[439,2422,2423],{"class":444},"            if",[439,2425,1428],{"class":444},[439,2427,1431],{"class":448},[439,2429,2430],{"class":98,"line":863},[439,2431,2432],{"class":444},"                continue\n",[439,2434,2435,2437],{"class":98,"line":868},[439,2436,742],{"class":444},[439,2438,593],{"class":448},[439,2440,2441],{"class":98,"line":886},[439,2442,2443],{"class":448},"                batch.append(Order.model_validate_json(line))\n",[439,2445,2446,2448,2451,2453],{"class":98,"line":902},[439,2447,775],{"class":444},[439,2449,2450],{"class":448}," ValidationError ",[439,2452,952],{"class":444},[439,2454,2455],{"class":448}," exc:\n",[439,2457,2458,2461,2463,2465,2467,2470,2472,2474,2476,2479,2481],{"class":98,"line":933},[439,2459,2460],{"class":444},"                raise",[439,2462,2326],{"class":448},[439,2464,2213],{"class":918},[439,2466,639],{"class":444},[439,2468,2469],{"class":505},"422",[439,2471,519],{"class":448},[439,2473,2338],{"class":918},[439,2475,639],{"class":444},[439,2477,2478],{"class":448},"exc.errors()[:",[439,2480,145],{"class":505},[439,2482,2483],{"class":448},"])\n",[439,2485,2486,2489,2491],{"class":98,"line":958},[439,2487,2488],{"class":448},"            accepted ",[439,2490,755],{"class":444},[439,2492,1466],{"class":505},[439,2494,2495,2497,2499,2501,2503,2505],{"class":98,"line":984},[439,2496,2423],{"class":444},[439,2498,722],{"class":505},[439,2500,1048],{"class":448},[439,2502,1051],{"class":444},[439,2504,1054],{"class":505},[439,2506,593],{"class":448},[439,2508,2509,2512],{"class":98,"line":990},[439,2510,2511],{"class":444},"                await",[439,2513,2514],{"class":448}," persist(batch)\n",[439,2516,2517],{"class":98,"line":1011},[439,2518,2519],{"class":448},"                batch.clear()\n",[439,2521,2522],{"class":98,"line":1034},[439,2523,478],{"emptyLinePlaceholder":477},[439,2525,2526,2528],{"class":98,"line":1040},[439,2527,1077],{"class":444},[439,2529,2530],{"class":448}," buf.strip():\n",[439,2532,2533,2536,2538],{"class":98,"line":1059},[439,2534,2535],{"class":448},"        batch.append(Order.model_validate_json(",[439,2537,619],{"class":505},[439,2539,2540],{"class":448},"(buf)))\n",[439,2542,2543,2546,2548],{"class":98,"line":1068},[439,2544,2545],{"class":448},"        accepted ",[439,2547,755],{"class":444},[439,2549,1466],{"class":505},[439,2551,2552,2554],{"class":98,"line":1074},[439,2553,1077],{"class":444},[439,2555,1080],{"class":448},[439,2557,2558,2560],{"class":98,"line":1083},[439,2559,1086],{"class":444},[439,2561,2514],{"class":448},[439,2563,2564,2566,2569,2572],{"class":98,"line":1091},[439,2565,1514],{"class":444},[439,2567,2568],{"class":448}," {",[439,2570,2571],{"class":515},"\"accepted\"",[439,2573,2574],{"class":448},": accepted}\n",[439,2576,2577],{"class":98,"line":1096},[439,2578,478],{"emptyLinePlaceholder":477},[439,2580,2581],{"class":98,"line":1101},[439,2582,478],{"emptyLinePlaceholder":477},[439,2584,2585,2587,2589,2592,2595,2597],{"class":98,"line":1122},[439,2586,871],{"class":444},[439,2588,677],{"class":444},[439,2590,2591],{"class":589}," persist",[439,2593,2594],{"class":448},"(batch: list[Order]) -> ",[439,2596,625],{"class":505},[439,2598,593],{"class":448},[439,2600,2601,2604],{"class":98,"line":1138},[439,2602,2603],{"class":505},"    ...",[439,2605,2606],{"class":929},"  # bulk insert; see the async SQLAlchemy guide\n",[14,2608,2609,2610,2613,2614,2618],{},"Three details make this production-safe. The byte counter enforces a limit yourself, because ",[18,2611,2612],{},"Content-Length"," is absent on a chunked upload and trusting the header is how you get a memory bomb. The 413 fires early, so an abusive client burns your bandwidth for a few seconds rather than your RAM for a minute. And a 202 with a count, not a 200 with the parsed data, keeps the response small — hand the heavy work to ",[23,2615,2617],{"href":2616},"\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002F","background jobs with Celery"," once you have durable rows on disk.",[64,2620,73,2624,73,2627,73,2630,73,2632,73,2639,73,2642,73,2646,73,2649,73,2653,73,2656,73,2660,73,2665,73,2667,73,2669,73,2674,73,2679,73,2681,73,2685,73,2689,73,2693,73,2697,73,2701,73,2704,73,2707,73,2709,73,2712,73,2716],{"viewBox":252,"role":67,"ariaLabelledBy":2621,"xmlns":71,"style":72},[2622,2623],"bigjson-seq-t","bigjson-seq-d",[75,2625,2626],{"id":2622},"Chunked upload sequence between client, API and database",[79,2628,2629],{"id":2623},"Sequence diagram: the client posts a chunked NDJSON body, the API reads chunks, validates lines, flushes batches of 5000 rows to Postgres, and returns 202 accepted or 413 when the byte limit is exceeded.",[83,2631],{"x":85,"y":85,"width":86,"height":264,"fill":88},[266,2633,268,2634,73],{},[270,2635,276,2637,268],{"id":2636,"viewBox":273,"refX":274,"refY":145,"markerWidth":130,"markerHeight":130,"orient":275},"bigjson-seq-arrow",[278,2638],{"d":280,"fill":169},[83,2640],{"x":2042,"y":2034,"width":358,"height":2641,"rx":288,"fill":289,"stroke":132,"style":306},"34",[90,2643,2645],{"x":2644,"y":101,"fill":94,"style":295},"95","client",[83,2647],{"x":2648,"y":2034,"width":358,"height":2641,"rx":288,"fill":289,"stroke":146,"style":306},"290",[90,2650,2652],{"x":2651,"y":101,"fill":94,"style":295},"355","FastAPI worker",[83,2654],{"x":2655,"y":2034,"width":358,"height":2641,"rx":288,"fill":289,"stroke":116,"style":306},"550",[90,2657,2659],{"x":2658,"y":101,"fill":94,"style":295},"615","Postgres",[98,2661],{"x1":2644,"y1":2662,"x2":2644,"y2":2663,"stroke":103,"style":2664},"54","272","stroke-width:1;stroke-dasharray:5 4;",[98,2666],{"x1":2651,"y1":2662,"x2":2651,"y2":2663,"stroke":103,"style":2664},[98,2668],{"x1":2658,"y1":2662,"x2":2658,"y2":2663,"stroke":103,"style":2664},[98,2670],{"x1":2644,"y1":2671,"x2":2672,"y2":2671,"stroke":169,"style":2673},"86","349","stroke-width:2;marker-end:url(#bigjson-seq-arrow);",[90,2675,2678],{"x":2676,"y":2677,"fill":94,"style":315},"222","78","POST \u002Fimports\u002Forders, chunked NDJSON",[98,2680],{"x1":2644,"y1":340,"x2":2672,"y2":340,"stroke":169,"style":2673},[90,2682,2684],{"x":2676,"y":2683,"fill":94,"style":315},"114","64 KB chunk, repeated",[83,2686],{"x":264,"y":1970,"width":2687,"height":2042,"rx":2688,"fill":88,"stroke":146,"style":290},"110","6",[90,2690,2692],{"x":2651,"y":2691,"fill":94,"style":315},"153","split + validate",[98,2694],{"x1":2651,"y1":2695,"x2":2696,"y2":2695,"stroke":169,"style":2673},"184","609",[90,2698,2700],{"x":2699,"y":355,"fill":94,"style":315},"482","flush 5000 rows",[98,2702],{"x1":2658,"y1":102,"x2":2703,"y2":102,"stroke":169,"style":2673},"361",[90,2705,2706],{"x":2699,"y":2012,"fill":94,"style":315},"commit ack",[98,2708],{"x1":2672,"y1":1984,"x2":1980,"y2":1984,"stroke":169,"style":2673},[90,2710,2711],{"x":2676,"y":168,"fill":94,"style":315},"202 accepted, count only",[90,2713,2715],{"x":2676,"y":2714,"fill":169,"style":315},"264","413 the moment the byte budget is exceeded",[90,2717,2719],{"x":92,"y":2718,"fill":169,"style":170},"288","Worker memory stays at one chunk plus one batch for the whole upload.",[14,2721,2722,2723,2726,2727,2730,2731,2734,2735,2739],{},"Serving large responses follows the same rule in reverse. Return a ",[18,2724,2725],{},"StreamingResponse"," over an async generator that yields one ",[18,2728,2729],{},"orjson.dumps(row) + b\"\\n\""," per row with media type ",[18,2732,2733],{},"application\u002Fx-ndjson",", and pull rows with a server-side cursor so the database does not materialize the result set either. The same generator mechanics power ",[23,2736,2738],{"href":2737},"\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fstreaming-llm-responses-through-fastapi\u002F","streaming LLM responses through FastAPI"," — different payload, identical backpressure story.",[40,2741],{},[43,2743,2745],{"id":2744},"when-to-choose-streaming-and-when-to-skip-it","When to Choose Streaming, and When to Skip It",[14,2747,2748,2749,2753],{},"Stream when any of these is true: a single payload can exceed roughly 50 MB; payload size is set by the customer rather than by you; your instance has under 1 GB of RAM; or the job needs to report progress while it runs. Under those conditions the cost model is decisive — a flat memory profile lets you run imports on the same small instance that serves traffic, which is the difference between one $7 service and a second $40 one that exists only for imports. Work through ",[23,2750,2752],{"href":2751},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fcalculating-cost-per-api-request\u002F","calculating cost per API request"," before you decide the bigger box is cheaper than the refactor; it usually is not.",[14,2755,2756,2757,2759],{},"Skip streaming when the payload is genuinely small, when you need the whole document in memory anyway (sorting across all records, resolving joins between distant parts of the tree), or when the JSON is deeply nested with no repeating record path for ",[18,2758,33],{}," to target. The wrong reason to stream is that it sounds more professional. Streaming code is harder to test, harder to reason about mid-batch, and it turns a single atomic parse into a partial-failure surface you now have to handle.",[64,2761,73,2766,73,2769,73,2772,73,2774,73,2780,73,2785,73,2788,73,2791,73,2795,73,2799,73,2802,73,2806,73,2809,73,2813,73,2816,73,2819,73,2823,73,2828,73,2831,73,2833,73,2836,73,2839,73,2842,73,2844,73,2848,73,2851,73,2854],{"viewBox":2762,"role":67,"ariaLabelledBy":2763,"xmlns":71,"style":72},"0 0 720 290",[2764,2765],"bigjson-tree-t","bigjson-tree-d",[75,2767,2768],{"id":2764},"Decision tree for choosing a JSON parsing strategy",[79,2770,2771],{"id":2765},"A decision tree: payloads under 50 MB use json.loads; larger payloads where you control the producer use NDJSON; otherwise use ijson, or stage to disk when the whole document is needed at once.",[83,2773],{"x":85,"y":85,"width":86,"height":2648,"fill":88},[83,2775],{"x":2776,"y":2777,"width":2778,"height":2779,"rx":288,"fill":289,"stroke":132,"style":306},"255","18","210","40",[90,2781,2784],{"x":2782,"y":2783,"fill":94,"style":295},"360","43","Payload over 50 MB?",[278,2786],{"d":2787,"fill":397,"stroke":103,"style":290},"M300 58 L300 84 L150 84 L150 106",[90,2789,2790],{"x":2676,"y":2677,"fill":169,"style":315},"no",[83,2792],{"x":2793,"y":2794,"width":2778,"height":2779,"rx":288,"fill":88,"stroke":146,"style":306},"45","106",[90,2796,2798],{"x":2008,"y":2797,"fill":94,"style":295},"131","json.loads, move on",[278,2800],{"d":2801,"fill":397,"stroke":103,"style":290},"M420 58 L420 84 L500 84 L500 106",[90,2803,2805],{"x":2804,"y":2677,"fill":169,"style":315},"470","yes",[83,2807],{"x":2808,"y":2794,"width":2778,"height":2779,"rx":288,"fill":289,"stroke":132,"style":306},"395",[90,2810,2812],{"x":2811,"y":2797,"fill":94,"style":295},"500","You control the producer?",[278,2814],{"d":2815,"fill":397,"stroke":103,"style":290},"M560 146 L560 168 L610 168 L610 190",[90,2817,2805],{"x":2818,"y":140,"fill":169,"style":315},"592",[83,2820],{"x":2821,"y":2822,"width":2822,"height":2779,"rx":288,"fill":88,"stroke":146,"style":306},"505","190",[90,2824,2827],{"x":2825,"y":2826,"fill":94,"style":295},"600","215","Emit NDJSON lines",[278,2829],{"d":2830,"fill":397,"stroke":103,"style":290},"M440 146 L440 168 L300 168 L300 190",[90,2832,2790],{"x":2029,"y":140,"fill":169,"style":315},[83,2834],{"x":2835,"y":2822,"width":2778,"height":2779,"rx":288,"fill":289,"stroke":132,"style":306},"195",[90,2837,2838],{"x":264,"y":2826,"fill":94,"style":295},"Need every record at once?",[278,2840],{"d":2841,"fill":397,"stroke":103,"style":290},"M250 230 L250 248 L150 248 L150 262",[90,2843,2805],{"x":366,"y":1984,"fill":169,"style":315},[90,2845,2847],{"x":2008,"y":2846,"fill":94,"style":295},"277","Stage rows to Postgres",[278,2849],{"d":2850,"fill":397,"stroke":103,"style":290},"M350 230 L350 248 L420 248 L420 262",[90,2852,2790],{"x":2853,"y":1984,"fill":169,"style":315},"392",[90,2855,2857],{"x":2856,"y":2846,"fill":94,"style":295},"420","ijson event stream",[40,2859],{},[43,2861,2863],{"id":2862},"the-migration-path","The Migration Path",[14,2865,2866,2867,2869,2870,2873],{},"Switching an existing import is a half-day job if you do it in this order. First, capture a real payload from production and record the baseline with the profiler above — you need the \"before\" number to defend the change. Second, wrap the parse in a generator function that yields records, leaving ",[18,2868,37],{}," inside it; every caller now consumes an iterator instead of a list, which is the only change that touches business logic. Third, swap the generator body for ",[18,2871,2872],{},"ijson.items_async"," or the NDJSON loop, since the call sites already expect a stream. Fourth, add the batch flush with a size from an env var so you can tune it in production without a deploy. Fifth, re-run the profiler, then drop the instance memory limit to just above the new peak so a regression fails loudly in staging instead of silently costing you money for months.",[14,2875,2876,2877,2880,2881,2885],{},"Two operational guardrails belong in the same pull request. Set ",[18,2878,2879],{},"read=None"," on the httpx timeout for the streaming call but keep the connect timeout tight, or a slow upstream will hang a worker indefinitely; combine that with ",[23,2882,2884],{"href":2883},"\u002Fgetting-started-with-python-apis-for-builders\u002Fmaking-http-requests-with-requests-library\u002Fretrying-failed-http-requests-with-tenacity\u002F","retrying failed HTTP requests with tenacity"," and record a resume checkpoint every batch so a retry does not restart 400 MB from byte zero. And log records-per-second plus peak RSS per job, because streaming failures are gradual — memory creeps when someone appends to a list they meant to clear, and only a metric catches it.",[40,2887],{},[43,2889,2891],{"id":2890},"builder-verdict","Builder Verdict",[14,2893,2894,2895,2897,2898,2901,2902,2904,2905,2907],{},"NDJSON wins. When you own both ends of the pipe, newline-delimited JSON beats every alternative on all three axes that matter to a small commercial API: it used 31 MB against ",[18,2896,37],{},"'s 3.1 GB, it finished ",[1931,2899,2900],{},"faster"," than the buffered parse rather than slower, and the consumer is a ",[18,2903,248],{}," loop any junior can debug at 2 a.m. It also degrades honestly, skipping one bad record instead of discarding a million good ones. When you do not own the producer — which is most third-party exports — ",[18,2906,33],{}," is the correct fallback, and the 2.3x slowdown against a buffered parse is a fair price for a 65x memory reduction and a first record in 0.3 seconds. Reach for the bigger instance only when your profiler says the whole document genuinely must be resident. Every other time, streaming keeps your import workload on the same small box that serves your traffic, and that single decision is often the difference between a 70% gross margin and a 40% one.",[43,2909,2911],{"id":2910},"faq","FAQ",[14,2913,2914,2918],{},[2915,2916,2917],"strong",{},"Does streaming actually reduce my hosting bill, or just my memory graph?","\nIt changes the instance class you can buy. Buffered parsing of a 420 MB export needs 4 GB of headroom, which is typically ten times the monthly cost of the 512 MB instance a streaming import runs on — and you pay for that headroom every hour, not just during the eleven seconds the job runs.",[14,2920,2921,2924],{},[2915,2922,2923],{},"Is ijson slower than json.loads, and does that hurt my throughput?","\nYes, roughly 2.3x slower on wall clock for a full pass, because it is a pure event parser rather than a C fast path over a complete buffer. It rarely hurts real throughput though: the import runs off the request path, and time to first record drops from 11.4 seconds to 0.3, so progress reporting and early failure detection both improve.",[14,2926,2927,2930,2931,2934],{},[2915,2928,2929],{},"What is the migration risk of moving a customer-facing endpoint to NDJSON?","\nLow if you version it. Keep the existing JSON-array endpoint alive, add the NDJSON one under a new path or content type, and let clients opt in with an ",[18,2932,2933],{},"Accept: application\u002Fx-ndjson"," header. Retire the array version only after your usage metrics show no traffic on it for a full billing cycle.",[14,2936,2937,2940,2941,2943],{},[2915,2938,2939],{},"How do I stop a customer from uploading a payload that kills my worker?","\nCount bytes yourself inside the stream loop and raise a 413 the instant you cross the limit, since a chunked upload has no trustworthy ",[18,2942,2612],{},". Set the ceiling per pricing tier from an env var, and log the rejection so sales can upsell rather than support having to explain a crash.",[14,2945,2946,2949],{},[2915,2947,2948],{},"Should the parsed records go straight into the database or into a queue?","\nInto the database, in batches, then queue the job id. Durable rows plus a 202 response means a client retry does not re-transfer hundreds of megabytes, and the expensive transformation work happens on a worker you can scale separately from your API.",[43,2951,2953],{"id":2952},"related","Related",[14,2955,2956],{},[2915,2957,2958],{},"Same track:",[2960,2961,2962,2968,2974],"ul",{},[2963,2964,2965,2967],"li",{},[23,2966,26],{"href":25}," — the parent guide covering defensive parsing end to end.",[2963,2969,2970,2973],{},[23,2971,2972],{"href":1174},"Validating JSON with Pydantic v2"," — how to validate each streamed record without slowing the loop.",[2963,2975,2976,2979],{},[23,2977,2978],{"href":426},"httpx vs requests for Async"," — why the async client is the right transport for long streaming bodies.",[14,2981,2982],{},[2915,2983,2984],{},"Other tracks:",[2960,2986,2987,2994],{},[2963,2988,2989,2993],{},[23,2990,2992],{"href":2991},"\u002Fscaling-and-operating-production-python-apis\u002Fasync-database-access-with-sqlalchemy\u002F","Async Database Access with SQLAlchemy"," — bulk-inserting the batches your parser emits.",[2963,2995,2996,3000],{},[23,2997,2999],{"href":2998},"\u002Fscaling-and-operating-production-python-apis\u002Fmonitoring-and-logging-python-apis\u002F","Monitoring and Logging Python APIs"," — catching the slow memory creep that streaming jobs are prone to.",[3002,3003,3004],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":435,"searchDepth":452,"depth":452,"links":3006},[3007,3008,3009,3010,3011,3012,3013,3014,3015],{"id":45,"depth":452,"text":46},{"id":176,"depth":452,"text":177},{"id":1557,"depth":452,"text":1558},{"id":2047,"depth":452,"text":2048},{"id":2744,"depth":452,"text":2745},{"id":2862,"depth":452,"text":2863},{"id":2890,"depth":452,"text":2891},{"id":2910,"depth":452,"text":2911},{"id":2952,"depth":452,"text":2953},"Parse multi-hundred-megabyte JSON in Python without exhausting RAM: ijson streaming, NDJSON, chunked FastAPI uploads, and the memory numbers that justify it.","md",{"pageTitle":3019,"type":3020,"datePublished":3021,"dateModified":3021},"Handling Large JSON Payloads with Streaming in Python","article","2026-07-23","\u002Fgetting-started-with-python-apis-for-builders\u002Fparsing-json-responses\u002Fhandling-large-json-payloads-with-streaming",{"title":5,"description":3016},"getting-started-with-python-apis-for-builders\u002Fparsing-json-responses\u002Fhandling-large-json-payloads-with-streaming\u002Findex","BH89cZUzdbuhOqbSXdSeyQvZ02biE6FYbvRhZFCakJU",{"@context":3027,"@type":3028,"mainEntity":3029},"https:\u002F\u002Fschema.org","FAQPage",[3030,3035,3038,3041,3044],{"@type":3031,"name":2917,"acceptedAnswer":3032},"Question",{"@type":3033,"text":3034},"Answer","It changes the instance class you can buy. Buffered parsing of a 420 MB export needs 4 GB of headroom, which is typically ten times the monthly cost of the 512 MB instance a streaming import runs on — and you pay for that headroom every hour, not just during the eleven seconds the job runs.",{"@type":3031,"name":2923,"acceptedAnswer":3036},{"@type":3033,"text":3037},"Yes, roughly 2.3x slower on wall clock for a full pass, because it is a pure event parser rather than a C fast path over a complete buffer. It rarely hurts real throughput though: the import runs off the request path, and time to first record drops from 11.4 seconds to 0.3, so progress reporting and early failure detection both improve.",{"@type":3031,"name":2929,"acceptedAnswer":3039},{"@type":3033,"text":3040},"Low if you version it. Keep the existing JSON-array endpoint alive, add the NDJSON one under a new path or content type, and let clients opt in with an Accept: application\u002Fx-ndjson header. Retire the array version only after your usage metrics show no traffic on it for a full billing cycle.",{"@type":3031,"name":2939,"acceptedAnswer":3042},{"@type":3033,"text":3043},"Count bytes yourself inside the stream loop and raise a 413 the instant you cross the limit, since a chunked upload has no trustworthy Content-Length. Set the ceiling per pricing tier from an env var, and log the rejection so sales can upsell rather than support having to explain a crash.",{"@type":3031,"name":2948,"acceptedAnswer":3045},{"@type":3033,"text":3046},"Into the database, in batches, then queue the job id. Durable rows plus a 202 response means a client retry does not re-transfer hundreds of megabytes, and the expensive transformation work happens on a worker you can scale separately from your API.",1784887028458]