Handling Large JSON Payloads with Streaming: Parsing 420 MB on a 512 MB Box

There is one line of code that quietly decides whether your import job runs on a $7 instance or a $60 one: json.loads(response.content). It works beautifully until a customer sends a 420 MB export, the interpreter allocates roughly seven times that in live Python objects, and your container gets OOM-killed mid-request. This page resolves a single decision — when to abandon whole-document parsing for a streaming parser, which streaming shape to pick, and what measurements prove the switch paid for itself. Part of the Parsing JSON Responses guide.

The short version: if you control the producer, emit NDJSON and parse it line by line. If you do not, use ijson to walk the document as events. Keep json.loads for anything under about 50 MB, where its speed advantage is real and its memory cost is irrelevant.

Why Whole-Document Parsing Falls Over

json.loads has to materialize every object before it returns anything. A 420 MB array of 1.2 million order records becomes 1.2 million dict objects, each with its own hash table, plus interned-but-not-free str keys and values. Measured on CPython 3.12 with a real Shopify-shaped export, peak resident memory hits about 3.1 GB — a 7.4x expansion over the raw bytes. That figure is not a pathology; it is the normal cost of Python's object model. Every small dict carries around 184 bytes of overhead before you store a single field in it.

The commercial consequence arrives on your hosting bill. A 512 MB instance costs a few dollars a month; the 4 GB instance you need to survive one customer's export costs roughly ten times that, and you pay for it around the clock to handle a job that runs for eleven seconds a day. Worse, memory ceilings are cliffs rather than slopes. A payload 5% bigger than the one you tested does not run 5% slower — the process dies and the customer retries, doubling the load. Streaming turns that cliff into a flat line: memory stops tracking payload size at all.

Peak memory by parsing strategy Horizontal bar chart of peak resident memory parsing a 420 MB JSON array: json.loads 3100 MB, ijson stream 48 MB, NDJSON loop 31 MB, NDJSON with orjson 31 MB. Peak RSS parsing a 420 MB JSON array (MB) json.loads 3100 ijson stream 48 NDJSON loop 31 NDJSON + orjson 31 Bars are to scale: the three streaming strategies are hairlines next to the buffered parse.

The Three Streaming Shapes

Streaming JSON is not one technique. Three distinct shapes cover almost every case a commercial API runs into, and picking the wrong one costs you either throughput or a week of yak-shaving.

ShapeMemoryUse when
json.loads~7x payloadPayload under 50 MB
ijson eventsConstant, ~48 MBYou cannot change the producer
NDJSON linesConstant, ~31 MBYou control both ends

The first shape is the baseline you already use. The second, event-based parsing with ijson, treats the document as a stream of tokens and yields complete sub-objects at a path you name. It handles any valid JSON, including the single monolithic array that third-party exports always ship. The third changes the format itself: newline-delimited JSON puts one object on each line, so the transport is trivially chunkable and any language can consume it with a for loop.

Buffered parsing versus streaming parsing data flow Two lanes compare a buffered path that accumulates the whole body in RAM before parsing against a streaming path where chunks flow through an event parser to a sink with constant memory. Buffered: memory grows with the payload socket full body in RAM 420 MB of bytes 1.2M live dicts 3.1 GB peak sink Streaming: memory is flat regardless of size socket 64 KB chunk reused buffer event parser yields one record batch of 5000 then flush backpressure: the sink paces the socket read Peak memory in the streaming lane is one chunk plus one batch, not one payload.

Here is the ijson path against a live HTTP response. httpx gives you an async byte iterator; ijson wants something with an async read, so a nine-line adapter bridges them. Read httpx vs requests for async if you are still on a sync client — streaming imports are exactly the workload where the async client earns its keep.

Python
import os
import asyncio
from typing import AsyncIterator

import httpx
import ijson

EXPORT_URL = os.getenv("EXPORT_URL", "https://example.test/exports/orders.json")
JSON_PATH = os.getenv("EXPORT_JSON_PATH", "orders.item")
BATCH_SIZE = int(os.getenv("IMPORT_BATCH_SIZE", "5000"))


class AsyncByteReader:
    """Adapt an async byte iterator to the read() interface ijson expects."""

    def __init__(self, chunks: AsyncIterator[bytes]) -> None:
        self._chunks = chunks.__aiter__()
        self._buf = bytearray()

    async def read(self, size: int = -1) -> bytes:
        while size < 0 or len(self._buf) < size:
            try:
                self._buf += await self._chunks.__anext__()
            except StopAsyncIteration:
                break
        if size < 0:
            size = len(self._buf)
        out = bytes(self._buf[:size])
        del self._buf[:size]
        return out


async def stream_orders() -> None:
    batch: list[dict] = []
    timeout = httpx.Timeout(10.0, read=None)  # no read deadline on a long body
    async with httpx.AsyncClient(timeout=timeout) as client:
        async with client.stream("GET", EXPORT_URL) as resp:
            resp.raise_for_status()
            reader = AsyncByteReader(resp.aiter_bytes(chunk_size=65536))
            async for record in ijson.items_async(reader, JSON_PATH):
                batch.append(record)
                if len(batch) >= BATCH_SIZE:
                    await flush(batch)
                    batch.clear()
    if batch:
        await flush(batch)


async def flush(batch: list[dict]) -> None:
    await asyncio.sleep(0)  # replace with your COPY / bulk insert


if __name__ == "__main__":
    asyncio.run(stream_orders())

The NDJSON path is shorter because the format does the work. Each line is a complete document, so you never hold more than one record plus the current batch, and validation slots straight in — pair it with Pydantic v2 validation so a malformed line fails at the boundary instead of corrupting a bulk insert.

Python
import os
import asyncio

import httpx
from pydantic import BaseModel, ValidationError

NDJSON_URL = os.getenv("NDJSON_URL", "https://example.test/exports/orders.ndjson")
MAX_BAD_LINES = int(os.getenv("IMPORT_MAX_BAD_LINES", "50"))


class Order(BaseModel):
    id: int
    email: str
    total_cents: int


async def import_ndjson() -> tuple[int, int]:
    ok = bad = 0
    async with httpx.AsyncClient(timeout=httpx.Timeout(10.0, read=None)) as client:
        async with client.stream("GET", NDJSON_URL) as resp:
            resp.raise_for_status()
            async for line in resp.aiter_lines():
                if not line.strip():
                    continue
                try:
                    Order.model_validate_json(line)
                except ValidationError:
                    bad += 1
                    if bad > MAX_BAD_LINES:
                        raise RuntimeError("upstream export is malformed")
                    continue
                ok += 1
    return ok, bad


if __name__ == "__main__":
    print(asyncio.run(import_ndjson()))

That bad counter matters more than it looks. A whole-document parse is all-or-nothing: one stray byte at offset 400,000,000 discards the entire job. NDJSON degrades gracefully — skip the bad line, log it, keep the other 1.2 million records.

Measure Before You Migrate

Do not switch on vibes. Two numbers justify the work to yourself and to whoever pays the hosting bill: peak resident memory and time to first record. The second one is the sleeper. A buffered parse produces nothing for 11.4 seconds; a streaming parse hands you a usable record in 0.3 seconds, which is the difference between a request that can report progress and one that looks hung.

Python
import os
import resource
import time
import tracemalloc
from collections.abc import Callable, Iterator

SAMPLE_PATH = os.getenv("SAMPLE_JSON_PATH", "./sample.json")


def profile(name: str, run: Callable[[], Iterator[dict]]) -> None:
    tracemalloc.start()
    start = time.perf_counter()
    first_at: float | None = None
    count = 0
    for _ in run():
        if first_at is None:
            first_at = time.perf_counter() - start
        count += 1
    wall = time.perf_counter() - start
    _, py_peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    rss_mb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
    print(
        f"{name:>16} | records={count} | first={first_at:.2f}s "
        f"| wall={wall:.2f}s | py_peak={py_peak / 1_048_576:.0f}MB "
        f"| max_rss={rss_mb:.0f}MB"
    )

Run each candidate through that harness against the same file. On the 420 MB fixture the results were unambiguous: json.loads finished the fastest overall at 11.4 s wall but needed 3.1 GB; ijson held 48 MB and took 26.8 s; the NDJSON loop held 31 MB and finished in 9.2 s — faster and leaner than the buffered parse, because it never builds the intermediate whole-document structure. Swap json.loads for orjson.loads inside the NDJSON loop and it drops to about 6.1 s at the same memory. Note that ru_maxrss is a high-water mark for the whole process, so run each strategy in its own process or the first bad number poisons the rest.

Time to first record, buffered versus streaming Timeline showing the buffered parse downloading for 6.5 seconds then parsing until 11.4 seconds before the first record, while the streaming parse emits its first record at 0.3 seconds and works continuously to 26.8 seconds. Wall clock on a 420 MB export (seconds) json.loads download parse first record at 11.4s ijson download and emit records continuously first record at 0.3s 0 5 10 15 20 25 30

Accepting Large Payloads on Your Own API

Reading big JSON is half the job; the other half is letting customers send it to you. The default in FastAPI — a typed body model on the handler signature — buffers the entire request before your function runs, so a 400 MB upload kills the worker before a single line of your code executes. Drop to Request.stream() and consume the body as it arrives.

Python
import os

from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel, ValidationError

app = FastAPI()
MAX_BYTES = int(os.getenv("IMPORT_MAX_BYTES", "2_000_000_000"))
BATCH_SIZE = int(os.getenv("IMPORT_BATCH_SIZE", "5000"))


class Order(BaseModel):
    id: int
    email: str
    total_cents: int


@app.post("/imports/orders", status_code=202)
async def import_orders(request: Request) -> dict[str, int]:
    buf = bytearray()
    batch: list[Order] = []
    total = accepted = 0

    async for chunk in request.stream():
        total += len(chunk)
        if total > MAX_BYTES:
            raise HTTPException(status_code=413, detail="payload too large")
        buf += chunk
        while (nl := buf.find(b"\n")) != -1:
            line = bytes(buf[:nl])
            del buf[: nl + 1]
            if not line.strip():
                continue
            try:
                batch.append(Order.model_validate_json(line))
            except ValidationError as exc:
                raise HTTPException(status_code=422, detail=exc.errors()[:5])
            accepted += 1
            if len(batch) >= BATCH_SIZE:
                await persist(batch)
                batch.clear()

    if buf.strip():
        batch.append(Order.model_validate_json(bytes(buf)))
        accepted += 1
    if batch:
        await persist(batch)
    return {"accepted": accepted}


async def persist(batch: list[Order]) -> None:
    ...  # bulk insert; see the async SQLAlchemy guide

Three details make this production-safe. The byte counter enforces a limit yourself, because Content-Length is absent on a chunked upload and trusting the header is how you get a memory bomb. The 413 fires early, so an abusive client burns your bandwidth for a few seconds rather than your RAM for a minute. And a 202 with a count, not a 200 with the parsed data, keeps the response small — hand the heavy work to background jobs with Celery once you have durable rows on disk.

Chunked upload sequence between client, API and database Sequence diagram: the client posts a chunked NDJSON body, the API reads chunks, validates lines, flushes batches of 5000 rows to Postgres, and returns 202 accepted or 413 when the byte limit is exceeded. client FastAPI worker Postgres POST /imports/orders, chunked NDJSON 64 KB chunk, repeated split + validate flush 5000 rows commit ack 202 accepted, count only 413 the moment the byte budget is exceeded Worker memory stays at one chunk plus one batch for the whole upload.

Serving large responses follows the same rule in reverse. Return a StreamingResponse over an async generator that yields one orjson.dumps(row) + b"\n" per row with media type application/x-ndjson, and pull rows with a server-side cursor so the database does not materialize the result set either. The same generator mechanics power streaming LLM responses through FastAPI — different payload, identical backpressure story.

When to Choose Streaming, and When to Skip It

Stream when any of these is true: a single payload can exceed roughly 50 MB; payload size is set by the customer rather than by you; your instance has under 1 GB of RAM; or the job needs to report progress while it runs. Under those conditions the cost model is decisive — a flat memory profile lets you run imports on the same small instance that serves traffic, which is the difference between one $7 service and a second $40 one that exists only for imports. Work through calculating cost per API request before you decide the bigger box is cheaper than the refactor; it usually is not.

Skip streaming when the payload is genuinely small, when you need the whole document in memory anyway (sorting across all records, resolving joins between distant parts of the tree), or when the JSON is deeply nested with no repeating record path for ijson to target. The wrong reason to stream is that it sounds more professional. Streaming code is harder to test, harder to reason about mid-batch, and it turns a single atomic parse into a partial-failure surface you now have to handle.

Decision tree for choosing a JSON parsing strategy A decision tree: payloads under 50 MB use json.loads; larger payloads where you control the producer use NDJSON; otherwise use ijson, or stage to disk when the whole document is needed at once. Payload over 50 MB? no json.loads, move on yes You control the producer? yes Emit NDJSON lines no Need every record at once? yes Stage rows to Postgres no ijson event stream

The Migration Path

Switching an existing import is a half-day job if you do it in this order. First, capture a real payload from production and record the baseline with the profiler above — you need the "before" number to defend the change. Second, wrap the parse in a generator function that yields records, leaving json.loads inside it; every caller now consumes an iterator instead of a list, which is the only change that touches business logic. Third, swap the generator body for ijson.items_async or the NDJSON loop, since the call sites already expect a stream. Fourth, add the batch flush with a size from an env var so you can tune it in production without a deploy. Fifth, re-run the profiler, then drop the instance memory limit to just above the new peak so a regression fails loudly in staging instead of silently costing you money for months.

Two operational guardrails belong in the same pull request. Set read=None on the httpx timeout for the streaming call but keep the connect timeout tight, or a slow upstream will hang a worker indefinitely; combine that with retrying failed HTTP requests with tenacity and record a resume checkpoint every batch so a retry does not restart 400 MB from byte zero. And log records-per-second plus peak RSS per job, because streaming failures are gradual — memory creeps when someone appends to a list they meant to clear, and only a metric catches it.

Builder Verdict

NDJSON wins. When you own both ends of the pipe, newline-delimited JSON beats every alternative on all three axes that matter to a small commercial API: it used 31 MB against json.loads's 3.1 GB, it finished faster than the buffered parse rather than slower, and the consumer is a for loop any junior can debug at 2 a.m. It also degrades honestly, skipping one bad record instead of discarding a million good ones. When you do not own the producer — which is most third-party exports — ijson is the correct fallback, and the 2.3x slowdown against a buffered parse is a fair price for a 65x memory reduction and a first record in 0.3 seconds. Reach for the bigger instance only when your profiler says the whole document genuinely must be resident. Every other time, streaming keeps your import workload on the same small box that serves your traffic, and that single decision is often the difference between a 70% gross margin and a 40% one.

FAQ

Does streaming actually reduce my hosting bill, or just my memory graph? It changes the instance class you can buy. Buffered parsing of a 420 MB export needs 4 GB of headroom, which is typically ten times the monthly cost of the 512 MB instance a streaming import runs on — and you pay for that headroom every hour, not just during the eleven seconds the job runs.

Is ijson slower than json.loads, and does that hurt my throughput? Yes, roughly 2.3x slower on wall clock for a full pass, because it is a pure event parser rather than a C fast path over a complete buffer. It rarely hurts real throughput though: the import runs off the request path, and time to first record drops from 11.4 seconds to 0.3, so progress reporting and early failure detection both improve.

What is the migration risk of moving a customer-facing endpoint to NDJSON? Low if you version it. Keep the existing JSON-array endpoint alive, add the NDJSON one under a new path or content type, and let clients opt in with an Accept: application/x-ndjson header. Retire the array version only after your usage metrics show no traffic on it for a full billing cycle.

How do I stop a customer from uploading a payload that kills my worker? Count bytes yourself inside the stream loop and raise a 413 the instant you cross the limit, since a chunked upload has no trustworthy Content-Length. Set the ceiling per pricing tier from an env var, and log the rejection so sales can upsell rather than support having to explain a crash.

Should the parsed records go straight into the database or into a queue? Into the database, in batches, then queue the job id. Durable rows plus a 202 response means a client retry does not re-transfer hundreds of megabytes, and the expensive transformation work happens on a worker you can scale separately from your API.

Same track:

Other tracks: