[{"data":1,"prerenderedAt":3551},["ShallowReactive",2],{"page-\u002Fbuilding-monetizing-api-driven-micro-saas\u002Ftracking-api-usage-and-analytics\u002Flogging-api-usage-events-to-postgres\u002F":3,"faq-schema-\u002Fbuilding-monetizing-api-driven-micro-saas\u002Ftracking-api-usage-and-analytics\u002Flogging-api-usage-events-to-postgres\u002F":3530},{"id":4,"title":5,"body":6,"description":3520,"extension":3521,"meta":3522,"navigation":345,"path":3526,"seo":3527,"stem":3528,"__hash__":3529},"content\u002Fbuilding-monetizing-api-driven-micro-saas\u002Ftracking-api-usage-and-analytics\u002Flogging-api-usage-events-to-postgres\u002Findex.md","Logging API Usage Events to Postgres",{"type":7,"value":8,"toc":3508},"minimark",[9,13,28,31,36,48,62,133,136,250,253,257,271,282,1119,1122,1355,1432,1436,1443,1465,1736,1752,1846,1850,1861,1869,2600,2620,2722,2726,2729,3272,3285,3382,3386,3394,3397,3401,3419,3423,3429,3433,3443,3449,3455,3461,3467,3471,3504],[10,11,5],"h1",{"id":12},"logging-api-usage-events-to-postgres",[14,15,16,17,22,23,27],"p",{},"Every commercial API eventually needs one table it can defend in front of a customer who says \"we never made 412,000 calls.\" That table is the usage event log, and where you put it decides whether your billing is auditable or merely plausible. Part of the ",[18,19,21],"a",{"href":20},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Ftracking-api-usage-and-analytics\u002F","Tracking API Usage and Analytics"," guide, this page resolves a single question: how do you make Postgres the durable record of every billable request without paying for that durability in p99 latency? The answer is a buffered writer using ",[24,25,26],"code",{},"COPY",", a schema built for per-key rollups rather than for row lookups, monthly partitions with an automated retention job, and a reconciliation query that catches drift before Stripe finalizes an invoice.",[14,29,30],{},"Resist reaching for a purpose-built analytics store on day one. Plain Postgres handles tens of millions of usage rows a month on a $30 instance, it already sits in your stack, and it joins usage against customers and subscriptions in one query. The engineering that matters is not the database choice; it is keeping the write off the hot path.",[32,33,35],"h2",{"id":34},"the-write-path-is-the-entire-decision","The write path is the entire decision",[14,37,38,39,42,43,47],{},"There are four realistic ways to get an event from a finished request into durable storage, and they differ by two orders of magnitude in what they cost the caller. A synchronous ",[24,40,41],{},"INSERT"," inside the request cycle adds a full round trip, a pooled connection, and a WAL flush to every response. On a modest managed Postgres that is roughly 4 ms at p99, and it competes for the same connections your product queries need — the fastest route to ",[18,44,46],{"href":45},"\u002Fscaling-and-operating-production-python-apis\u002Fasync-database-access-with-sqlalchemy\u002Ffixing-connection-pool-exhaustion\u002F","connection pool exhaustion"," that exists.",[14,49,50,51,53,54,57,58,61],{},"Buffering in process and flushing batches with ",[24,52,26],{}," costs the request a ",[24,55,56],{},"put_nowait"," call on an ",[24,59,60],{},"asyncio.Queue",": about 40 microseconds, no I\u002FO, no connection. The trade is a loss window — events sitting in memory when the process dies are gone. A Redis Stream buffer closes that window at the cost of a network hop per request and a second moving part; a stdout log plus a shipper closes it too, but now you own a log pipeline and your billing data lives somewhere you cannot join against.",[63,64,65,81],"table",{},[66,67,68],"thead",{},[69,70,71,75,78],"tr",{},[72,73,74],"th",{},"Write path",[72,76,77],{},"Added p99",[72,79,80],{},"Loss window",[82,83,84,98,111,122],"tbody",{},[69,85,86,92,95],{},[87,88,89,90],"td",{},"Inline ",[24,91,41],{},[87,93,94],{},"~4.1 ms",[87,96,97],{},"none",[69,99,100,105,108],{},[87,101,102,103],{},"Buffer + ",[24,104,26],{},[87,106,107],{},"~0.04 ms",[87,109,110],{},"one flush interval",[69,112,113,116,119],{},[87,114,115],{},"Redis Stream",[87,117,118],{},"~0.4 ms",[87,120,121],{},"Redis fsync policy",[69,123,124,127,130],{},[87,125,126],{},"stdout + shipper",[87,128,129],{},"~0.1 ms",[87,131,132],{},"shipper backlog",[14,134,135],{},"Look at where those milliseconds land in a real request budget. A monetized endpoint doing external work spends 35 to 40 ms in the handler, so 4 ms of billing I\u002FO raises p99 by ten percent and adds a failure mode: if Postgres is slow, your API is slow, for a write nobody is waiting on.",[137,138,146,147,146,151,146,155,146,162,146,170,146,175,146,182,146,187,146,192,146,197,146,201,146,204,146,206,146,209,146,214,146,218,146,225,146,228,146,232,146,235,146,239,146,246],"svg",{"viewBox":139,"role":140,"ariaLabelledBy":141,"xmlns":144,"style":145},"0 0 720 250","img",[142,143],"luep-budget-t","luep-budget-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;height:auto;margin:1.5rem 0;font-family:var(--font-sans);","\n  ",[148,149,150],"title",{"id":142},"Request latency budget with inline insert versus buffered writer",[152,153,154],"desc",{"id":143},"Two stacked bars showing a 41.3 millisecond request when the usage row is inserted inline, versus 37.2 milliseconds when the event is buffered in memory and flushed with COPY.",[156,157],"rect",{"x":158,"y":158,"width":159,"height":160,"fill":161},"0","720","250","var(--c-surface)",[163,164,169],"text",{"x":165,"y":166,"fill":167,"style":168},"20","34","var(--c-text)","font-size:13;font-family:var(--font-sans);","p99 for one billable request",[163,171,174],{"x":165,"y":172,"fill":167,"style":173},"76","font-size:12;font-family:var(--font-sans);","Inline INSERT",[156,176],{"x":177,"y":178,"width":179,"height":180,"fill":181},"150","60","14","30","var(--c-teal)",[156,183],{"x":184,"y":178,"width":185,"height":180,"fill":186},"164","432","var(--c-blue)",[156,188],{"x":189,"y":178,"width":190,"height":180,"fill":191},"596","49","var(--c-coral)",[163,193,196],{"x":194,"y":195,"fill":167,"style":173},"655","80","41.3 ms",[163,198,200],{"x":165,"y":199,"fill":167,"style":173},"136","Buffer + COPY",[156,202],{"x":177,"y":203,"width":179,"height":180,"fill":181},"120",[156,205],{"x":184,"y":203,"width":185,"height":180,"fill":186},[156,207],{"x":189,"y":203,"width":208,"height":180,"fill":191},"4",[163,210,213],{"x":211,"y":212,"fill":167,"style":173},"610","140","37.2 ms",[156,215],{"x":177,"y":216,"width":217,"height":217,"fill":181},"182","12",[163,219,224],{"x":220,"y":221,"fill":222,"style":223},"168","192","var(--c-text-muted)","font-size:11;font-family:var(--font-sans);","auth + middleware 1.2 ms",[156,226],{"x":227,"y":216,"width":217,"height":217,"fill":186},"340",[163,229,231],{"x":230,"y":221,"fill":222,"style":223},"358","handler work 36 ms",[156,233],{"x":234,"y":216,"width":217,"height":217,"fill":191},"510",[163,236,238],{"x":237,"y":221,"fill":222,"style":223},"528","usage write",[240,241],"line",{"x1":165,"y1":242,"x2":243,"y2":242,"stroke":244,"style":245},"214","700","var(--c-border)","stroke-width:1;",[163,247,249],{"x":165,"y":248,"fill":222,"style":223},"234","Buffering removes 4.06 ms and one pooled connection from every request.",[14,251,252],{},"Pick the buffered writer. The loss window is bounded by your flush interval, and you can shrink it to a second or two without meaningfully changing throughput. Losing two seconds of usage events during an unplanned pod kill costs you a few cents; adding 4 ms to every request costs you a slower API forever and a support ticket the first time your database has a bad afternoon.",[32,254,256],{"id":255},"buffered-writes-with-copy-not-executemany","Buffered writes with COPY, not executemany",[14,258,259,262,263,265,266,270],{},[24,260,261],{},"copy_records_to_table"," on asyncpg pushes rows through the Postgres binary COPY protocol. It is roughly five to ten times faster than a parameterized multi-row ",[24,264,41],{}," for batches of a few hundred, and it takes one connection for one transaction regardless of batch size. If you have not picked a driver yet, the ",[18,267,269],{"href":268},"\u002Fscaling-and-operating-production-python-apis\u002Fasync-database-access-with-sqlalchemy\u002Fasyncpg-vs-psycopg3-for-fastapi\u002F","asyncpg vs psycopg3 comparison"," covers the trade — for this workload asyncpg wins on raw COPY throughput.",[14,272,273,274,277,278,281],{},"The drain loop below collects up to ",[24,275,276],{},"USAGE_BATCH_MAX"," rows or waits ",[24,279,280],{},"USAGE_FLUSH_SECONDS",", whichever comes first, then writes once. Two details matter commercially. First, a failed flush spools to disk instead of vanishing, so a database restart costs you a delay, not revenue. Second, the shutdown handler drains the queue before the process exits, which turns every planned deploy into a zero-loss event.",[283,284,289],"pre",{"className":285,"code":286,"language":287,"meta":288,"style":288},"language-python shiki shiki-themes github-light github-dark","# usage\u002Fwriter.py\nimport asyncio\nimport json\nimport os\nfrom pathlib import Path\n\nimport asyncpg\n\nBATCH_MAX = int(os.getenv(\"USAGE_BATCH_MAX\", \"500\"))\nFLUSH_SECONDS = float(os.getenv(\"USAGE_FLUSH_SECONDS\", \"2.0\"))\nQUEUE_MAXSIZE = int(os.getenv(\"USAGE_QUEUE_MAXSIZE\", \"20000\"))\nSPOOL_DIR = Path(os.getenv(\"USAGE_SPOOL_DIR\", \"\u002Fvar\u002Ftmp\u002Fusage-spool\"))\nDSN = os.getenv(\"USAGE_DATABASE_URL\", \"postgresql:\u002F\u002Flocalhost\u002Fapi\")\n\nCOLUMNS = [\n    \"occurred_at\", \"customer_id\", \"api_key_id\", \"route\",\n    \"status_code\", \"duration_ms\", \"billable_units\",\n]\n\nqueue: asyncio.Queue[tuple] = asyncio.Queue(maxsize=QUEUE_MAXSIZE)\n\n\nasync def _collect_batch() -> list[tuple]:\n    \"\"\"Block for the first row, then fill the batch until it is full or the timer expires.\"\"\"\n    loop = asyncio.get_running_loop()\n    batch = [await queue.get()]\n    deadline = loop.time() + FLUSH_SECONDS\n    while len(batch) \u003C BATCH_MAX:\n        remaining = deadline - loop.time()\n        if remaining \u003C= 0:\n            break\n        try:\n            batch.append(await asyncio.wait_for(queue.get(), remaining))\n        except TimeoutError:  # builtin alias since 3.11\n            break\n    return batch\n\n\ndef _spool(batch: list[tuple]) -> None:\n    SPOOL_DIR.mkdir(parents=True, exist_ok=True)\n    path = SPOOL_DIR \u002F f\"{os.getpid()}-{id(batch)}.jsonl\"\n    with path.open(\"w\", encoding=\"utf-8\") as fh:\n        for row in batch:\n            fh.write(json.dumps(row, default=str) + \"\\n\")\n\n\nasync def drain_forever(pool: asyncpg.Pool) -> None:\n    while True:\n        batch = await _collect_batch()\n        try:\n            async with pool.acquire() as conn:\n                await conn.copy_records_to_table(\n                    \"usage_event\", records=batch, columns=COLUMNS\n                )\n        except (asyncpg.PostgresError, OSError):\n            _spool(batch)          # replay later; never lose a billable row\n            await asyncio.sleep(1.0)\n        finally:\n            for _ in batch:\n                queue.task_done()\n","python","",[24,290,291,299,310,318,326,340,347,355,360,389,412,434,455,477,482,493,517,535,541,546,574,579,584,605,611,622,639,656,677,694,711,717,725,736,751,756,765,770,775,797,825,867,898,913,941,946,951,968,978,992,999,1016,1025,1049,1055,1069,1078,1092,1100,1113],{"__ignoreMap":288},[292,293,295],"span",{"class":240,"line":294},1,[292,296,298],{"class":297},"sJ8bj","# usage\u002Fwriter.py\n",[292,300,302,306],{"class":240,"line":301},2,[292,303,305],{"class":304},"szBVR","import",[292,307,309],{"class":308},"sVt8B"," asyncio\n",[292,311,313,315],{"class":240,"line":312},3,[292,314,305],{"class":304},[292,316,317],{"class":308}," json\n",[292,319,321,323],{"class":240,"line":320},4,[292,322,305],{"class":304},[292,324,325],{"class":308}," os\n",[292,327,329,332,335,337],{"class":240,"line":328},5,[292,330,331],{"class":304},"from",[292,333,334],{"class":308}," pathlib ",[292,336,305],{"class":304},[292,338,339],{"class":308}," Path\n",[292,341,343],{"class":240,"line":342},6,[292,344,346],{"emptyLinePlaceholder":345},true,"\n",[292,348,350,352],{"class":240,"line":349},7,[292,351,305],{"class":304},[292,353,354],{"class":308}," asyncpg\n",[292,356,358],{"class":240,"line":357},8,[292,359,346],{"emptyLinePlaceholder":345},[292,361,363,367,370,373,376,380,383,386],{"class":240,"line":362},9,[292,364,366],{"class":365},"sj4cs","BATCH_MAX",[292,368,369],{"class":304}," =",[292,371,372],{"class":365}," int",[292,374,375],{"class":308},"(os.getenv(",[292,377,379],{"class":378},"sZZnC","\"USAGE_BATCH_MAX\"",[292,381,382],{"class":308},", ",[292,384,385],{"class":378},"\"500\"",[292,387,388],{"class":308},"))\n",[292,390,392,395,397,400,402,405,407,410],{"class":240,"line":391},10,[292,393,394],{"class":365},"FLUSH_SECONDS",[292,396,369],{"class":304},[292,398,399],{"class":365}," float",[292,401,375],{"class":308},[292,403,404],{"class":378},"\"USAGE_FLUSH_SECONDS\"",[292,406,382],{"class":308},[292,408,409],{"class":378},"\"2.0\"",[292,411,388],{"class":308},[292,413,415,418,420,422,424,427,429,432],{"class":240,"line":414},11,[292,416,417],{"class":365},"QUEUE_MAXSIZE",[292,419,369],{"class":304},[292,421,372],{"class":365},[292,423,375],{"class":308},[292,425,426],{"class":378},"\"USAGE_QUEUE_MAXSIZE\"",[292,428,382],{"class":308},[292,430,431],{"class":378},"\"20000\"",[292,433,388],{"class":308},[292,435,437,440,442,445,448,450,453],{"class":240,"line":436},12,[292,438,439],{"class":365},"SPOOL_DIR",[292,441,369],{"class":304},[292,443,444],{"class":308}," Path(os.getenv(",[292,446,447],{"class":378},"\"USAGE_SPOOL_DIR\"",[292,449,382],{"class":308},[292,451,452],{"class":378},"\"\u002Fvar\u002Ftmp\u002Fusage-spool\"",[292,454,388],{"class":308},[292,456,458,461,463,466,469,471,474],{"class":240,"line":457},13,[292,459,460],{"class":365},"DSN",[292,462,369],{"class":304},[292,464,465],{"class":308}," os.getenv(",[292,467,468],{"class":378},"\"USAGE_DATABASE_URL\"",[292,470,382],{"class":308},[292,472,473],{"class":378},"\"postgresql:\u002F\u002Flocalhost\u002Fapi\"",[292,475,476],{"class":308},")\n",[292,478,480],{"class":240,"line":479},14,[292,481,346],{"emptyLinePlaceholder":345},[292,483,485,488,490],{"class":240,"line":484},15,[292,486,487],{"class":365},"COLUMNS",[292,489,369],{"class":304},[292,491,492],{"class":308}," [\n",[292,494,496,499,501,504,506,509,511,514],{"class":240,"line":495},16,[292,497,498],{"class":378},"    \"occurred_at\"",[292,500,382],{"class":308},[292,502,503],{"class":378},"\"customer_id\"",[292,505,382],{"class":308},[292,507,508],{"class":378},"\"api_key_id\"",[292,510,382],{"class":308},[292,512,513],{"class":378},"\"route\"",[292,515,516],{"class":308},",\n",[292,518,520,523,525,528,530,533],{"class":240,"line":519},17,[292,521,522],{"class":378},"    \"status_code\"",[292,524,382],{"class":308},[292,526,527],{"class":378},"\"duration_ms\"",[292,529,382],{"class":308},[292,531,532],{"class":378},"\"billable_units\"",[292,534,516],{"class":308},[292,536,538],{"class":240,"line":537},18,[292,539,540],{"class":308},"]\n",[292,542,544],{"class":240,"line":543},19,[292,545,346],{"emptyLinePlaceholder":345},[292,547,549,552,555,558,561,564,568,570,572],{"class":240,"line":548},20,[292,550,551],{"class":308},"queue: asyncio.Queue[",[292,553,554],{"class":365},"tuple",[292,556,557],{"class":308},"] ",[292,559,560],{"class":304},"=",[292,562,563],{"class":308}," asyncio.Queue(",[292,565,567],{"class":566},"s4XuR","maxsize",[292,569,560],{"class":304},[292,571,417],{"class":365},[292,573,476],{"class":308},[292,575,577],{"class":240,"line":576},21,[292,578,346],{"emptyLinePlaceholder":345},[292,580,582],{"class":240,"line":581},22,[292,583,346],{"emptyLinePlaceholder":345},[292,585,587,590,593,597,600,602],{"class":240,"line":586},23,[292,588,589],{"class":304},"async",[292,591,592],{"class":304}," def",[292,594,596],{"class":595},"sScJk"," _collect_batch",[292,598,599],{"class":308},"() -> list[",[292,601,554],{"class":365},[292,603,604],{"class":308},"]:\n",[292,606,608],{"class":240,"line":607},24,[292,609,610],{"class":378},"    \"\"\"Block for the first row, then fill the batch until it is full or the timer expires.\"\"\"\n",[292,612,614,617,619],{"class":240,"line":613},25,[292,615,616],{"class":308},"    loop ",[292,618,560],{"class":304},[292,620,621],{"class":308}," asyncio.get_running_loop()\n",[292,623,625,628,630,633,636],{"class":240,"line":624},26,[292,626,627],{"class":308},"    batch ",[292,629,560],{"class":304},[292,631,632],{"class":308}," [",[292,634,635],{"class":304},"await",[292,637,638],{"class":308}," queue.get()]\n",[292,640,642,645,647,650,653],{"class":240,"line":641},27,[292,643,644],{"class":308},"    deadline ",[292,646,560],{"class":304},[292,648,649],{"class":308}," loop.time() ",[292,651,652],{"class":304},"+",[292,654,655],{"class":365}," FLUSH_SECONDS\n",[292,657,659,662,665,668,671,674],{"class":240,"line":658},28,[292,660,661],{"class":304},"    while",[292,663,664],{"class":365}," len",[292,666,667],{"class":308},"(batch) ",[292,669,670],{"class":304},"\u003C",[292,672,673],{"class":365}," BATCH_MAX",[292,675,676],{"class":308},":\n",[292,678,680,683,685,688,691],{"class":240,"line":679},29,[292,681,682],{"class":308},"        remaining ",[292,684,560],{"class":304},[292,686,687],{"class":308}," deadline ",[292,689,690],{"class":304},"-",[292,692,693],{"class":308}," loop.time()\n",[292,695,697,700,703,706,709],{"class":240,"line":696},30,[292,698,699],{"class":304},"        if",[292,701,702],{"class":308}," remaining ",[292,704,705],{"class":304},"\u003C=",[292,707,708],{"class":365}," 0",[292,710,676],{"class":308},[292,712,714],{"class":240,"line":713},31,[292,715,716],{"class":304},"            break\n",[292,718,720,723],{"class":240,"line":719},32,[292,721,722],{"class":304},"        try",[292,724,676],{"class":308},[292,726,728,731,733],{"class":240,"line":727},33,[292,729,730],{"class":308},"            batch.append(",[292,732,635],{"class":304},[292,734,735],{"class":308}," asyncio.wait_for(queue.get(), remaining))\n",[292,737,739,742,745,748],{"class":240,"line":738},34,[292,740,741],{"class":304},"        except",[292,743,744],{"class":365}," TimeoutError",[292,746,747],{"class":308},":  ",[292,749,750],{"class":297},"# builtin alias since 3.11\n",[292,752,754],{"class":240,"line":753},35,[292,755,716],{"class":304},[292,757,759,762],{"class":240,"line":758},36,[292,760,761],{"class":304},"    return",[292,763,764],{"class":308}," batch\n",[292,766,768],{"class":240,"line":767},37,[292,769,346],{"emptyLinePlaceholder":345},[292,771,773],{"class":240,"line":772},38,[292,774,346],{"emptyLinePlaceholder":345},[292,776,778,781,784,787,789,792,795],{"class":240,"line":777},39,[292,779,780],{"class":304},"def",[292,782,783],{"class":595}," _spool",[292,785,786],{"class":308},"(batch: list[",[292,788,554],{"class":365},[292,790,791],{"class":308},"]) -> ",[292,793,794],{"class":365},"None",[292,796,676],{"class":308},[292,798,800,803,806,809,811,814,816,819,821,823],{"class":240,"line":799},40,[292,801,802],{"class":365},"    SPOOL_DIR",[292,804,805],{"class":308},".mkdir(",[292,807,808],{"class":566},"parents",[292,810,560],{"class":304},[292,812,813],{"class":365},"True",[292,815,382],{"class":308},[292,817,818],{"class":566},"exist_ok",[292,820,560],{"class":304},[292,822,813],{"class":365},[292,824,476],{"class":308},[292,826,828,831,833,836,839,842,845,848,851,854,856,859,862,864],{"class":240,"line":827},41,[292,829,830],{"class":308},"    path ",[292,832,560],{"class":304},[292,834,835],{"class":365}," SPOOL_DIR",[292,837,838],{"class":304}," \u002F",[292,840,841],{"class":304}," f",[292,843,844],{"class":378},"\"",[292,846,847],{"class":365},"{",[292,849,850],{"class":308},"os.getpid()",[292,852,853],{"class":365},"}",[292,855,690],{"class":378},[292,857,858],{"class":365},"{id",[292,860,861],{"class":308},"(batch)",[292,863,853],{"class":365},[292,865,866],{"class":378},".jsonl\"\n",[292,868,870,873,876,879,881,884,886,889,892,895],{"class":240,"line":869},42,[292,871,872],{"class":304},"    with",[292,874,875],{"class":308}," path.open(",[292,877,878],{"class":378},"\"w\"",[292,880,382],{"class":308},[292,882,883],{"class":566},"encoding",[292,885,560],{"class":304},[292,887,888],{"class":378},"\"utf-8\"",[292,890,891],{"class":308},") ",[292,893,894],{"class":304},"as",[292,896,897],{"class":308}," fh:\n",[292,899,901,904,907,910],{"class":240,"line":900},43,[292,902,903],{"class":304},"        for",[292,905,906],{"class":308}," row ",[292,908,909],{"class":304},"in",[292,911,912],{"class":308}," batch:\n",[292,914,916,919,922,924,927,929,931,934,937,939],{"class":240,"line":915},44,[292,917,918],{"class":308},"            fh.write(json.dumps(row, ",[292,920,921],{"class":566},"default",[292,923,560],{"class":304},[292,925,926],{"class":365},"str",[292,928,891],{"class":308},[292,930,652],{"class":304},[292,932,933],{"class":378}," \"",[292,935,936],{"class":365},"\\n",[292,938,844],{"class":378},[292,940,476],{"class":308},[292,942,944],{"class":240,"line":943},45,[292,945,346],{"emptyLinePlaceholder":345},[292,947,949],{"class":240,"line":948},46,[292,950,346],{"emptyLinePlaceholder":345},[292,952,954,956,958,961,964,966],{"class":240,"line":953},47,[292,955,589],{"class":304},[292,957,592],{"class":304},[292,959,960],{"class":595}," drain_forever",[292,962,963],{"class":308},"(pool: asyncpg.Pool) -> ",[292,965,794],{"class":365},[292,967,676],{"class":308},[292,969,971,973,976],{"class":240,"line":970},48,[292,972,661],{"class":304},[292,974,975],{"class":365}," True",[292,977,676],{"class":308},[292,979,981,984,986,989],{"class":240,"line":980},49,[292,982,983],{"class":308},"        batch ",[292,985,560],{"class":304},[292,987,988],{"class":304}," await",[292,990,991],{"class":308}," _collect_batch()\n",[292,993,995,997],{"class":240,"line":994},50,[292,996,722],{"class":304},[292,998,676],{"class":308},[292,1000,1002,1005,1008,1011,1013],{"class":240,"line":1001},51,[292,1003,1004],{"class":304},"            async",[292,1006,1007],{"class":304}," with",[292,1009,1010],{"class":308}," pool.acquire() ",[292,1012,894],{"class":304},[292,1014,1015],{"class":308}," conn:\n",[292,1017,1019,1022],{"class":240,"line":1018},52,[292,1020,1021],{"class":304},"                await",[292,1023,1024],{"class":308}," conn.copy_records_to_table(\n",[292,1026,1028,1031,1033,1036,1038,1041,1044,1046],{"class":240,"line":1027},53,[292,1029,1030],{"class":378},"                    \"usage_event\"",[292,1032,382],{"class":308},[292,1034,1035],{"class":566},"records",[292,1037,560],{"class":304},[292,1039,1040],{"class":308},"batch, ",[292,1042,1043],{"class":566},"columns",[292,1045,560],{"class":304},[292,1047,1048],{"class":365},"COLUMNS\n",[292,1050,1052],{"class":240,"line":1051},54,[292,1053,1054],{"class":308},"                )\n",[292,1056,1058,1060,1063,1066],{"class":240,"line":1057},55,[292,1059,741],{"class":304},[292,1061,1062],{"class":308}," (asyncpg.PostgresError, ",[292,1064,1065],{"class":365},"OSError",[292,1067,1068],{"class":308},"):\n",[292,1070,1072,1075],{"class":240,"line":1071},56,[292,1073,1074],{"class":308},"            _spool(batch)          ",[292,1076,1077],{"class":297},"# replay later; never lose a billable row\n",[292,1079,1081,1084,1087,1090],{"class":240,"line":1080},57,[292,1082,1083],{"class":304},"            await",[292,1085,1086],{"class":308}," asyncio.sleep(",[292,1088,1089],{"class":365},"1.0",[292,1091,476],{"class":308},[292,1093,1095,1098],{"class":240,"line":1094},58,[292,1096,1097],{"class":304},"        finally",[292,1099,676],{"class":308},[292,1101,1103,1106,1109,1111],{"class":240,"line":1102},59,[292,1104,1105],{"class":304},"            for",[292,1107,1108],{"class":308}," _ ",[292,1110,909],{"class":304},[292,1112,912],{"class":308},[292,1114,1116],{"class":240,"line":1115},60,[292,1117,1118],{"class":308},"                queue.task_done()\n",[14,1120,1121],{},"Wire the loop to the application lifespan so it finishes its work before the process leaves. Give it a dedicated pool of two connections so a flush can never starve product traffic.",[283,1123,1125],{"className":285,"code":1124,"language":287,"meta":288,"style":288},"# usage\u002Flifespan.py\nimport asyncio\nimport os\nfrom contextlib import asynccontextmanager\n\nimport asyncpg\nfrom fastapi import FastAPI\n\nfrom usage.writer import DSN, drain_forever, queue\n\nDRAIN_TIMEOUT = float(os.getenv(\"USAGE_DRAIN_TIMEOUT\", \"10.0\"))\n\n\n@asynccontextmanager\nasync def lifespan(app: FastAPI):\n    pool = await asyncpg.create_pool(DSN, min_size=1, max_size=2)\n    task = asyncio.create_task(drain_forever(pool))\n    try:\n        yield\n    finally:\n        try:\n            await asyncio.wait_for(queue.join(), DRAIN_TIMEOUT)\n        except TimeoutError:\n            pass\n        task.cancel()\n        await pool.close()\n",[24,1126,1127,1132,1138,1144,1156,1160,1166,1178,1182,1197,1201,1222,1226,1230,1235,1247,1283,1293,1300,1305,1312,1318,1329,1337,1342,1347],{"__ignoreMap":288},[292,1128,1129],{"class":240,"line":294},[292,1130,1131],{"class":297},"# usage\u002Flifespan.py\n",[292,1133,1134,1136],{"class":240,"line":301},[292,1135,305],{"class":304},[292,1137,309],{"class":308},[292,1139,1140,1142],{"class":240,"line":312},[292,1141,305],{"class":304},[292,1143,325],{"class":308},[292,1145,1146,1148,1151,1153],{"class":240,"line":320},[292,1147,331],{"class":304},[292,1149,1150],{"class":308}," contextlib ",[292,1152,305],{"class":304},[292,1154,1155],{"class":308}," asynccontextmanager\n",[292,1157,1158],{"class":240,"line":328},[292,1159,346],{"emptyLinePlaceholder":345},[292,1161,1162,1164],{"class":240,"line":342},[292,1163,305],{"class":304},[292,1165,354],{"class":308},[292,1167,1168,1170,1173,1175],{"class":240,"line":349},[292,1169,331],{"class":304},[292,1171,1172],{"class":308}," fastapi ",[292,1174,305],{"class":304},[292,1176,1177],{"class":308}," FastAPI\n",[292,1179,1180],{"class":240,"line":357},[292,1181,346],{"emptyLinePlaceholder":345},[292,1183,1184,1186,1189,1191,1194],{"class":240,"line":362},[292,1185,331],{"class":304},[292,1187,1188],{"class":308}," usage.writer ",[292,1190,305],{"class":304},[292,1192,1193],{"class":365}," DSN",[292,1195,1196],{"class":308},", drain_forever, queue\n",[292,1198,1199],{"class":240,"line":391},[292,1200,346],{"emptyLinePlaceholder":345},[292,1202,1203,1206,1208,1210,1212,1215,1217,1220],{"class":240,"line":414},[292,1204,1205],{"class":365},"DRAIN_TIMEOUT",[292,1207,369],{"class":304},[292,1209,399],{"class":365},[292,1211,375],{"class":308},[292,1213,1214],{"class":378},"\"USAGE_DRAIN_TIMEOUT\"",[292,1216,382],{"class":308},[292,1218,1219],{"class":378},"\"10.0\"",[292,1221,388],{"class":308},[292,1223,1224],{"class":240,"line":436},[292,1225,346],{"emptyLinePlaceholder":345},[292,1227,1228],{"class":240,"line":457},[292,1229,346],{"emptyLinePlaceholder":345},[292,1231,1232],{"class":240,"line":479},[292,1233,1234],{"class":595},"@asynccontextmanager\n",[292,1236,1237,1239,1241,1244],{"class":240,"line":484},[292,1238,589],{"class":304},[292,1240,592],{"class":304},[292,1242,1243],{"class":595}," lifespan",[292,1245,1246],{"class":308},"(app: FastAPI):\n",[292,1248,1249,1252,1254,1256,1259,1261,1263,1266,1268,1271,1273,1276,1278,1281],{"class":240,"line":495},[292,1250,1251],{"class":308},"    pool ",[292,1253,560],{"class":304},[292,1255,988],{"class":304},[292,1257,1258],{"class":308}," asyncpg.create_pool(",[292,1260,460],{"class":365},[292,1262,382],{"class":308},[292,1264,1265],{"class":566},"min_size",[292,1267,560],{"class":304},[292,1269,1270],{"class":365},"1",[292,1272,382],{"class":308},[292,1274,1275],{"class":566},"max_size",[292,1277,560],{"class":304},[292,1279,1280],{"class":365},"2",[292,1282,476],{"class":308},[292,1284,1285,1288,1290],{"class":240,"line":519},[292,1286,1287],{"class":308},"    task ",[292,1289,560],{"class":304},[292,1291,1292],{"class":308}," asyncio.create_task(drain_forever(pool))\n",[292,1294,1295,1298],{"class":240,"line":537},[292,1296,1297],{"class":304},"    try",[292,1299,676],{"class":308},[292,1301,1302],{"class":240,"line":543},[292,1303,1304],{"class":304},"        yield\n",[292,1306,1307,1310],{"class":240,"line":548},[292,1308,1309],{"class":304},"    finally",[292,1311,676],{"class":308},[292,1313,1314,1316],{"class":240,"line":576},[292,1315,722],{"class":304},[292,1317,676],{"class":308},[292,1319,1320,1322,1325,1327],{"class":240,"line":581},[292,1321,1083],{"class":304},[292,1323,1324],{"class":308}," asyncio.wait_for(queue.join(), ",[292,1326,1205],{"class":365},[292,1328,476],{"class":308},[292,1330,1331,1333,1335],{"class":240,"line":586},[292,1332,741],{"class":304},[292,1334,744],{"class":365},[292,1336,676],{"class":308},[292,1338,1339],{"class":240,"line":607},[292,1340,1341],{"class":304},"            pass\n",[292,1343,1344],{"class":240,"line":613},[292,1345,1346],{"class":308},"        task.cancel()\n",[292,1348,1349,1352],{"class":240,"line":624},[292,1350,1351],{"class":304},"        await",[292,1353,1354],{"class":308}," pool.close()\n",[137,1356,146,1361,146,1364,146,1367,146,1370,146,1373,146,1378,146,1380,146,1386,146,1388,146,1393,146,1398,146,1401,146,1404,146,1409,146,1411,146,1414,146,1418,146,1421,146,1424,146,1428],{"viewBox":1357,"role":140,"ariaLabelledBy":1358,"xmlns":144,"style":145},"0 0 720 290",[1359,1360],"luep-drain-t","luep-drain-d",[148,1362,1363],{"id":1359},"Queue depth over time with two-second COPY flushes",[152,1365,1366],{"id":1360},"Queue depth climbs as requests enqueue events and drops to zero at each two-second flush, with a final drain triggered by SIGTERM at five seconds.",[156,1368],{"x":158,"y":158,"width":159,"height":1369,"fill":161},"290",[163,1371,1372],{"x":165,"y":180,"fill":167,"style":168},"Queue depth between flushes (250 req\u002Fs)",[240,1374],{"x1":178,"y1":1375,"x2":1376,"y2":1375,"stroke":244,"style":1377},"210","670","stroke-width:1.5;",[240,1379],{"x1":178,"y1":178,"x2":178,"y2":1375,"stroke":244,"style":1377},[163,1381,1385],{"x":1382,"y":1383,"fill":222,"style":1384},"52","66","text-anchor:end;font-size:11;font-family:var(--font-sans);","500",[163,1387,158],{"x":1382,"y":242,"fill":222,"style":1384},[1389,1390],"polyline",{"points":1391,"fill":97,"stroke":181,"style":1392},"60,210 300,62 300,210 540,62 540,210 630,154 630,210","stroke-width:2.5;",[240,1394],{"x1":1395,"y1":1382,"x2":1395,"y2":1396,"stroke":191,"style":1397},"300","216","stroke-width:1.5;stroke-dasharray:4 4;",[240,1399],{"x1":1400,"y1":1382,"x2":1400,"y2":1396,"stroke":191,"style":1397},"540",[240,1402],{"x1":1403,"y1":1382,"x2":1403,"y2":1396,"stroke":186,"style":1397},"630",[163,1405,1408],{"x":1395,"y":1406,"fill":167,"style":1407},"46","text-anchor:middle;font-size:11;font-family:var(--font-sans);","COPY 500 rows",[163,1410,1408],{"x":1400,"y":1406,"fill":167,"style":1407},[163,1412,1413],{"x":1403,"y":1406,"fill":167,"style":1407},"SIGTERM drain",[163,1415,1417],{"x":178,"y":1416,"fill":222,"style":1407},"232","0 s",[163,1419,1420],{"x":1395,"y":1416,"fill":222,"style":1407},"2 s",[163,1422,1423],{"x":1400,"y":1416,"fill":222,"style":1407},"4 s",[163,1425,1427],{"x":1426,"y":1416,"fill":222,"style":1407},"660","5 s",[163,1429,1431],{"x":165,"y":1430,"fill":222,"style":223},"266","Worst-case loss equals one flush interval; the shutdown hook removes it on planned deploys.",[32,1433,1435],{"id":1434},"a-schema-shaped-for-rollups-not-lookups","A schema shaped for rollups, not lookups",[14,1437,1438,1439,1442],{},"Nobody ever reads a single usage event. Every query you will actually run is an aggregate over a customer and a time range, so design for sequential scans of recent data and for one covering index. Skip the surrogate primary key entirely — a ",[24,1440,1441],{},"bigint"," identity column costs eight bytes and a sequence hit per row and buys you nothing, because you will never fetch a usage event by id.",[14,1444,1445,1446,1449,1450,1453,1454,1457,1458,1461,1462,1464],{},"Two indexes carry the whole workload. A BRIN index on ",[24,1447,1448],{},"occurred_at"," is the star: on naturally time-ordered data it summarizes ranges of pages instead of storing a pointer per row, so it occupies a few hundred kilobytes where a btree would take gigabytes, and it barely slows the write. A btree on ",[24,1451,1452],{},"(customer_id, occurred_at)"," with ",[24,1455,1456],{},"billable_units"," in an ",[24,1459,1460],{},"INCLUDE"," clause serves the per-key rollup as an index-only scan. That is it. Every extra index you add is a tax on every ",[24,1463,26],{},".",[283,1466,1468],{"className":285,"code":1467,"language":287,"meta":288,"style":288},"# usage\u002Fschema.py\nimport asyncio\nimport os\n\nimport asyncpg\n\nDDL = \"\"\"\nCREATE TABLE IF NOT EXISTS usage_event (\n    occurred_at     timestamptz NOT NULL,\n    customer_id     text        NOT NULL,\n    api_key_id      text        NOT NULL,\n    route           text        NOT NULL,\n    status_code     smallint    NOT NULL,\n    duration_ms     integer     NOT NULL,\n    billable_units  integer     NOT NULL DEFAULT 1\n) PARTITION BY RANGE (occurred_at);\n\nCREATE INDEX IF NOT EXISTS ix_usage_brin_time\n    ON usage_event USING brin (occurred_at) WITH (pages_per_range = 32);\n\nCREATE INDEX IF NOT EXISTS ix_usage_customer_time\n    ON usage_event (customer_id, occurred_at) INCLUDE (billable_units);\n\nCREATE TABLE IF NOT EXISTS usage_rollup_daily (\n    day             date NOT NULL,\n    customer_id     text NOT NULL,\n    route           text NOT NULL,\n    calls           bigint NOT NULL,\n    billable_units  bigint NOT NULL,\n    PRIMARY KEY (day, customer_id, route)\n);\n\"\"\"\n\n\nasync def main() -> None:\n    conn = await asyncpg.connect(os.getenv(\"USAGE_DATABASE_URL\", \"postgresql:\u002F\u002Flocalhost\u002Fapi\"))\n    try:\n        await conn.execute(DDL)\n    finally:\n        await conn.close()\n\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n",[24,1469,1470,1475,1481,1487,1491,1497,1501,1511,1516,1521,1526,1531,1536,1541,1546,1551,1556,1560,1565,1570,1574,1579,1584,1588,1593,1598,1603,1608,1613,1618,1623,1628,1633,1637,1641,1657,1677,1683,1694,1700,1707,1711,1715,1731],{"__ignoreMap":288},[292,1471,1472],{"class":240,"line":294},[292,1473,1474],{"class":297},"# usage\u002Fschema.py\n",[292,1476,1477,1479],{"class":240,"line":301},[292,1478,305],{"class":304},[292,1480,309],{"class":308},[292,1482,1483,1485],{"class":240,"line":312},[292,1484,305],{"class":304},[292,1486,325],{"class":308},[292,1488,1489],{"class":240,"line":320},[292,1490,346],{"emptyLinePlaceholder":345},[292,1492,1493,1495],{"class":240,"line":328},[292,1494,305],{"class":304},[292,1496,354],{"class":308},[292,1498,1499],{"class":240,"line":342},[292,1500,346],{"emptyLinePlaceholder":345},[292,1502,1503,1506,1508],{"class":240,"line":349},[292,1504,1505],{"class":365},"DDL",[292,1507,369],{"class":304},[292,1509,1510],{"class":378}," \"\"\"\n",[292,1512,1513],{"class":240,"line":357},[292,1514,1515],{"class":378},"CREATE TABLE IF NOT EXISTS usage_event (\n",[292,1517,1518],{"class":240,"line":362},[292,1519,1520],{"class":378},"    occurred_at     timestamptz NOT NULL,\n",[292,1522,1523],{"class":240,"line":391},[292,1524,1525],{"class":378},"    customer_id     text        NOT NULL,\n",[292,1527,1528],{"class":240,"line":414},[292,1529,1530],{"class":378},"    api_key_id      text        NOT NULL,\n",[292,1532,1533],{"class":240,"line":436},[292,1534,1535],{"class":378},"    route           text        NOT NULL,\n",[292,1537,1538],{"class":240,"line":457},[292,1539,1540],{"class":378},"    status_code     smallint    NOT NULL,\n",[292,1542,1543],{"class":240,"line":479},[292,1544,1545],{"class":378},"    duration_ms     integer     NOT NULL,\n",[292,1547,1548],{"class":240,"line":484},[292,1549,1550],{"class":378},"    billable_units  integer     NOT NULL DEFAULT 1\n",[292,1552,1553],{"class":240,"line":495},[292,1554,1555],{"class":378},") PARTITION BY RANGE (occurred_at);\n",[292,1557,1558],{"class":240,"line":519},[292,1559,346],{"emptyLinePlaceholder":345},[292,1561,1562],{"class":240,"line":537},[292,1563,1564],{"class":378},"CREATE INDEX IF NOT EXISTS ix_usage_brin_time\n",[292,1566,1567],{"class":240,"line":543},[292,1568,1569],{"class":378},"    ON usage_event USING brin (occurred_at) WITH (pages_per_range = 32);\n",[292,1571,1572],{"class":240,"line":548},[292,1573,346],{"emptyLinePlaceholder":345},[292,1575,1576],{"class":240,"line":576},[292,1577,1578],{"class":378},"CREATE INDEX IF NOT EXISTS ix_usage_customer_time\n",[292,1580,1581],{"class":240,"line":581},[292,1582,1583],{"class":378},"    ON usage_event (customer_id, occurred_at) INCLUDE (billable_units);\n",[292,1585,1586],{"class":240,"line":586},[292,1587,346],{"emptyLinePlaceholder":345},[292,1589,1590],{"class":240,"line":607},[292,1591,1592],{"class":378},"CREATE TABLE IF NOT EXISTS usage_rollup_daily (\n",[292,1594,1595],{"class":240,"line":613},[292,1596,1597],{"class":378},"    day             date NOT NULL,\n",[292,1599,1600],{"class":240,"line":624},[292,1601,1602],{"class":378},"    customer_id     text NOT NULL,\n",[292,1604,1605],{"class":240,"line":641},[292,1606,1607],{"class":378},"    route           text NOT NULL,\n",[292,1609,1610],{"class":240,"line":658},[292,1611,1612],{"class":378},"    calls           bigint NOT NULL,\n",[292,1614,1615],{"class":240,"line":679},[292,1616,1617],{"class":378},"    billable_units  bigint NOT NULL,\n",[292,1619,1620],{"class":240,"line":696},[292,1621,1622],{"class":378},"    PRIMARY KEY (day, customer_id, route)\n",[292,1624,1625],{"class":240,"line":713},[292,1626,1627],{"class":378},");\n",[292,1629,1630],{"class":240,"line":719},[292,1631,1632],{"class":378},"\"\"\"\n",[292,1634,1635],{"class":240,"line":727},[292,1636,346],{"emptyLinePlaceholder":345},[292,1638,1639],{"class":240,"line":738},[292,1640,346],{"emptyLinePlaceholder":345},[292,1642,1643,1645,1647,1650,1653,1655],{"class":240,"line":753},[292,1644,589],{"class":304},[292,1646,592],{"class":304},[292,1648,1649],{"class":595}," main",[292,1651,1652],{"class":308},"() -> ",[292,1654,794],{"class":365},[292,1656,676],{"class":308},[292,1658,1659,1662,1664,1666,1669,1671,1673,1675],{"class":240,"line":758},[292,1660,1661],{"class":308},"    conn ",[292,1663,560],{"class":304},[292,1665,988],{"class":304},[292,1667,1668],{"class":308}," asyncpg.connect(os.getenv(",[292,1670,468],{"class":378},[292,1672,382],{"class":308},[292,1674,473],{"class":378},[292,1676,388],{"class":308},[292,1678,1679,1681],{"class":240,"line":767},[292,1680,1297],{"class":304},[292,1682,676],{"class":308},[292,1684,1685,1687,1690,1692],{"class":240,"line":772},[292,1686,1351],{"class":304},[292,1688,1689],{"class":308}," conn.execute(",[292,1691,1505],{"class":365},[292,1693,476],{"class":308},[292,1695,1696,1698],{"class":240,"line":777},[292,1697,1309],{"class":304},[292,1699,676],{"class":308},[292,1701,1702,1704],{"class":240,"line":799},[292,1703,1351],{"class":304},[292,1705,1706],{"class":308}," conn.close()\n",[292,1708,1709],{"class":240,"line":827},[292,1710,346],{"emptyLinePlaceholder":345},[292,1712,1713],{"class":240,"line":869},[292,1714,346],{"emptyLinePlaceholder":345},[292,1716,1717,1720,1723,1726,1729],{"class":240,"line":900},[292,1718,1719],{"class":304},"if",[292,1721,1722],{"class":365}," __name__",[292,1724,1725],{"class":304}," ==",[292,1727,1728],{"class":378}," \"__main__\"",[292,1730,676],{"class":308},[292,1732,1733],{"class":240,"line":915},[292,1734,1735],{"class":308},"    asyncio.run(main())\n",[14,1737,1738,1739,1742,1743,1746,1747,1751],{},"Resist adding a unique constraint for deduplication. On a partitioned table any unique index must include the partition key, so you would be enforcing uniqueness on ",[24,1740,1741],{},"(occurred_at, request_id)"," — which does not stop a replay with a different timestamp, and it forces a btree lookup on every inserted row. Deduplicate at rollup time with ",[24,1744,1745],{},"DISTINCT ON"," instead, where it costs you nothing per request. The same reasoning drives ",[18,1748,1750],{"href":1749},"\u002Fautomating-side-hustle-operations-with-apis\u002Fprocessing-webhooks-with-python\u002Fbuilding-an-idempotent-webhook-receiver\u002F","idempotent webhook receivers",": push uniqueness to the cheapest place that still gives a correct answer.",[137,1753,146,1758,146,1761,146,1764,146,1767,146,1774,146,1778,146,1782,146,1785,146,1788,146,1792,146,1798,146,1801,146,1803,146,1807,146,1810,146,1812,146,1815,146,1818,146,1822,146,1824,146,1829,146,1834,146,1838,146,1842],{"viewBox":1754,"role":140,"ariaLabelledBy":1755,"xmlns":144,"style":145},"0 0 720 320",[1756,1757],"luep-idx-t","luep-idx-d",[148,1759,1760],{"id":1756},"Which index serves which usage query",[152,1762,1763],{"id":1757},"A per-customer billing rollup uses the covering btree index on customer and time, while a whole-fleet latency scan uses the BRIN index on occurred_at, both reading the partitioned usage_event table.",[156,1765],{"x":158,"y":158,"width":159,"height":1766,"fill":161},"320",[156,1768],{"x":165,"y":1769,"width":1770,"height":1771,"rx":1772,"fill":1773,"stroke":244,"style":1377},"40","200","64","8","var(--c-surface-alt)",[163,1775,1777],{"x":203,"y":1383,"fill":167,"style":1776},"text-anchor:middle;font-size:12;font-family:var(--font-sans);","Billing rollup query",[163,1779,1781],{"x":203,"y":1780,"fill":222,"style":1407},"86","one customer, one month",[156,1783],{"x":165,"y":1784,"width":1770,"height":1771,"rx":1772,"fill":1773,"stroke":244,"style":1377},"190",[163,1786,1787],{"x":203,"y":1396,"fill":167,"style":1776},"Latency report query",[163,1789,1791],{"x":203,"y":1790,"fill":222,"style":1407},"236","all keys, last 24 h",[240,1793],{"x1":1794,"y1":1795,"x2":1796,"y2":1795,"stroke":222,"style":1797},"220","72","288","stroke-width:2;",[240,1799],{"x1":1794,"y1":1800,"x2":1796,"y2":1800,"stroke":222,"style":1797},"222",[156,1802],{"x":1369,"y":1769,"width":1770,"height":1771,"rx":1772,"fill":161,"stroke":186,"style":1797},[163,1804,1806],{"x":1805,"y":1383,"fill":167,"style":1776},"390","btree (customer, time)",[163,1808,1809],{"x":1805,"y":1780,"fill":222,"style":1407},"INCLUDE billable_units",[156,1811],{"x":1369,"y":1784,"width":1770,"height":1771,"rx":1772,"fill":161,"stroke":181,"style":1797},[163,1813,1814],{"x":1805,"y":1396,"fill":167,"style":1776},"BRIN (occurred_at)",[163,1816,1817],{"x":1805,"y":1790,"fill":222,"style":1407},"pages_per_range 32",[240,1819],{"x1":1820,"y1":1795,"x2":1821,"y2":212,"stroke":186,"style":1797},"490","556",[240,1823],{"x1":1820,"y1":1800,"x2":1821,"y2":184,"stroke":181,"style":1797},[156,1825],{"x":1826,"y":1827,"width":1828,"height":1795,"rx":1772,"fill":161,"stroke":244,"style":1377},"558","118","142",[163,1830,1833],{"x":1831,"y":1832,"fill":167,"style":1776},"629","146","usage_event",[163,1835,1837],{"x":1831,"y":1836,"fill":222,"style":1407},"166","7 columns, no PK",[163,1839,1841],{"x":1805,"y":1840,"fill":222,"style":1407},"128","index-only scan, no heap fetch",[163,1843,1845],{"x":1805,"y":1844,"fill":222,"style":1407},"286","summary index: ~300 KB per 10 M rows",[32,1847,1849],{"id":1848},"partitions-and-retention-that-stay-boring","Partitions and retention that stay boring",[14,1851,1852,1853,1856,1857,1860],{},"Partition by month with ",[24,1854,1855],{},"RANGE (occurred_at)"," and create partitions ahead of need. The payoff arrives on retention day: dropping a month of raw events becomes a metadata operation that finishes in milliseconds, while the equivalent ",[24,1858,1859],{},"DELETE"," rewrites gigabytes, bloats the write-ahead log, and leaves dead tuples for autovacuum to chew through during your busiest week.",[14,1862,1863,1864,1868],{},"Keep raw events for 90 days and daily rollups for 25 months. Ninety days covers a full billing cycle plus a dispute window; two years of rollups costs less than a coffee. Run the maintenance job daily through whatever scheduler you already have — a container cron entry or the ",[18,1865,1867],{"href":1866},"\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002F","Celery beat setup"," you use for other periodic work.",[283,1870,1872],{"className":285,"code":1871,"language":287,"meta":288,"style":288},"# usage\u002Fpartitions.py\nimport asyncio\nimport os\nfrom datetime import date, timedelta\n\nimport asyncpg\n\nMONTHS_AHEAD = int(os.getenv(\"USAGE_PARTITION_MONTHS_AHEAD\", \"2\"))\nRAW_RETENTION_DAYS = int(os.getenv(\"USAGE_RAW_RETENTION_DAYS\", \"90\"))\n\n\ndef _month_start(d: date, offset: int = 0) -> date:\n    month_index = d.year * 12 + (d.month - 1) + offset\n    return date(month_index \u002F\u002F 12, month_index % 12 + 1, 1)\n\n\nasync def ensure_partitions(conn: asyncpg.Connection, today: date) -> list[str]:\n    created = []\n    for offset in range(0, MONTHS_AHEAD + 1):\n        start = _month_start(today, offset)\n        end = _month_start(today, offset + 1)\n        name = f\"usage_event_{start:%Y_%m}\"\n        await conn.execute(\n            f\"CREATE TABLE IF NOT EXISTS {name} PARTITION OF usage_event \"\n            f\"FOR VALUES FROM ('{start.isoformat()}') TO ('{end.isoformat()}')\"\n        )\n        created.append(name)\n    return created\n\n\nasync def drop_expired(conn: asyncpg.Connection, today: date) -> list[str]:\n    cutoff = _month_start(today - timedelta(days=RAW_RETENTION_DAYS))\n    rows = await conn.fetch(\n        \"SELECT c.relname FROM pg_class c \"\n        \"JOIN pg_inherits i ON i.inhrelid = c.oid \"\n        \"JOIN pg_class p ON p.oid = i.inhparent \"\n        \"WHERE p.relname = 'usage_event' ORDER BY c.relname\"\n    )\n    dropped = []\n    for row in rows:\n        name = row[\"relname\"]\n        stamp = name.removeprefix(\"usage_event_\")\n        year, month = (int(part) for part in stamp.split(\"_\"))\n        if date(year, month, 1) \u003C cutoff:\n            await conn.execute(f\"ALTER TABLE usage_event DETACH PARTITION {name}\")\n            await conn.execute(f\"DROP TABLE {name}\")\n            dropped.append(name)\n    return dropped\n\n\nasync def main() -> None:\n    conn = await asyncpg.connect(os.getenv(\"USAGE_DATABASE_URL\", \"postgresql:\u002F\u002Flocalhost\u002Fapi\"))\n    try:\n        today = date.today()\n        print({\"created\": await ensure_partitions(conn, today),\n               \"dropped\": await drop_expired(conn, today)})\n    finally:\n        await conn.close()\n\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n",[24,1873,1874,1879,1885,1891,1903,1907,1913,1917,1938,1959,1963,1967,1987,2021,2051,2055,2059,2075,2085,2113,2123,2139,2171,2178,2196,2223,2228,2233,2240,2244,2248,2263,2287,2299,2304,2309,2314,2319,2324,2333,2344,2358,2373,2404,2420,2442,2463,2468,2475,2479,2483,2497,2515,2521,2531,2550,2562,2568,2574,2578,2582,2595],{"__ignoreMap":288},[292,1875,1876],{"class":240,"line":294},[292,1877,1878],{"class":297},"# usage\u002Fpartitions.py\n",[292,1880,1881,1883],{"class":240,"line":301},[292,1882,305],{"class":304},[292,1884,309],{"class":308},[292,1886,1887,1889],{"class":240,"line":312},[292,1888,305],{"class":304},[292,1890,325],{"class":308},[292,1892,1893,1895,1898,1900],{"class":240,"line":320},[292,1894,331],{"class":304},[292,1896,1897],{"class":308}," datetime ",[292,1899,305],{"class":304},[292,1901,1902],{"class":308}," date, timedelta\n",[292,1904,1905],{"class":240,"line":328},[292,1906,346],{"emptyLinePlaceholder":345},[292,1908,1909,1911],{"class":240,"line":342},[292,1910,305],{"class":304},[292,1912,354],{"class":308},[292,1914,1915],{"class":240,"line":349},[292,1916,346],{"emptyLinePlaceholder":345},[292,1918,1919,1922,1924,1926,1928,1931,1933,1936],{"class":240,"line":357},[292,1920,1921],{"class":365},"MONTHS_AHEAD",[292,1923,369],{"class":304},[292,1925,372],{"class":365},[292,1927,375],{"class":308},[292,1929,1930],{"class":378},"\"USAGE_PARTITION_MONTHS_AHEAD\"",[292,1932,382],{"class":308},[292,1934,1935],{"class":378},"\"2\"",[292,1937,388],{"class":308},[292,1939,1940,1943,1945,1947,1949,1952,1954,1957],{"class":240,"line":362},[292,1941,1942],{"class":365},"RAW_RETENTION_DAYS",[292,1944,369],{"class":304},[292,1946,372],{"class":365},[292,1948,375],{"class":308},[292,1950,1951],{"class":378},"\"USAGE_RAW_RETENTION_DAYS\"",[292,1953,382],{"class":308},[292,1955,1956],{"class":378},"\"90\"",[292,1958,388],{"class":308},[292,1960,1961],{"class":240,"line":391},[292,1962,346],{"emptyLinePlaceholder":345},[292,1964,1965],{"class":240,"line":414},[292,1966,346],{"emptyLinePlaceholder":345},[292,1968,1969,1971,1974,1977,1980,1982,1984],{"class":240,"line":436},[292,1970,780],{"class":304},[292,1972,1973],{"class":595}," _month_start",[292,1975,1976],{"class":308},"(d: date, offset: ",[292,1978,1979],{"class":365},"int",[292,1981,369],{"class":304},[292,1983,708],{"class":365},[292,1985,1986],{"class":308},") -> date:\n",[292,1988,1989,1992,1994,1997,2000,2003,2006,2009,2011,2014,2016,2018],{"class":240,"line":457},[292,1990,1991],{"class":308},"    month_index ",[292,1993,560],{"class":304},[292,1995,1996],{"class":308}," d.year ",[292,1998,1999],{"class":304},"*",[292,2001,2002],{"class":365}," 12",[292,2004,2005],{"class":304}," +",[292,2007,2008],{"class":308}," (d.month ",[292,2010,690],{"class":304},[292,2012,2013],{"class":365}," 1",[292,2015,891],{"class":308},[292,2017,652],{"class":304},[292,2019,2020],{"class":308}," offset\n",[292,2022,2023,2025,2028,2031,2033,2036,2039,2041,2043,2045,2047,2049],{"class":240,"line":479},[292,2024,761],{"class":304},[292,2026,2027],{"class":308}," date(month_index ",[292,2029,2030],{"class":304},"\u002F\u002F",[292,2032,2002],{"class":365},[292,2034,2035],{"class":308},", month_index ",[292,2037,2038],{"class":304},"%",[292,2040,2002],{"class":365},[292,2042,2005],{"class":304},[292,2044,2013],{"class":365},[292,2046,382],{"class":308},[292,2048,1270],{"class":365},[292,2050,476],{"class":308},[292,2052,2053],{"class":240,"line":484},[292,2054,346],{"emptyLinePlaceholder":345},[292,2056,2057],{"class":240,"line":495},[292,2058,346],{"emptyLinePlaceholder":345},[292,2060,2061,2063,2065,2068,2071,2073],{"class":240,"line":519},[292,2062,589],{"class":304},[292,2064,592],{"class":304},[292,2066,2067],{"class":595}," ensure_partitions",[292,2069,2070],{"class":308},"(conn: asyncpg.Connection, today: date) -> list[",[292,2072,926],{"class":365},[292,2074,604],{"class":308},[292,2076,2077,2080,2082],{"class":240,"line":537},[292,2078,2079],{"class":308},"    created ",[292,2081,560],{"class":304},[292,2083,2084],{"class":308}," []\n",[292,2086,2087,2090,2093,2095,2098,2101,2103,2105,2107,2109,2111],{"class":240,"line":543},[292,2088,2089],{"class":304},"    for",[292,2091,2092],{"class":308}," offset ",[292,2094,909],{"class":304},[292,2096,2097],{"class":365}," range",[292,2099,2100],{"class":308},"(",[292,2102,158],{"class":365},[292,2104,382],{"class":308},[292,2106,1921],{"class":365},[292,2108,2005],{"class":304},[292,2110,2013],{"class":365},[292,2112,1068],{"class":308},[292,2114,2115,2118,2120],{"class":240,"line":548},[292,2116,2117],{"class":308},"        start ",[292,2119,560],{"class":304},[292,2121,2122],{"class":308}," _month_start(today, offset)\n",[292,2124,2125,2128,2130,2133,2135,2137],{"class":240,"line":576},[292,2126,2127],{"class":308},"        end ",[292,2129,560],{"class":304},[292,2131,2132],{"class":308}," _month_start(today, offset ",[292,2134,652],{"class":304},[292,2136,2013],{"class":365},[292,2138,476],{"class":308},[292,2140,2141,2144,2146,2148,2151,2153,2156,2158,2161,2163,2166,2168],{"class":240,"line":581},[292,2142,2143],{"class":308},"        name ",[292,2145,560],{"class":304},[292,2147,841],{"class":304},[292,2149,2150],{"class":378},"\"usage_event_",[292,2152,847],{"class":365},[292,2154,2155],{"class":308},"start:",[292,2157,2038],{"class":304},[292,2159,2160],{"class":308},"Y_",[292,2162,2038],{"class":304},[292,2164,2165],{"class":308},"m",[292,2167,853],{"class":365},[292,2169,2170],{"class":378},"\"\n",[292,2172,2173,2175],{"class":240,"line":586},[292,2174,1351],{"class":304},[292,2176,2177],{"class":308}," conn.execute(\n",[292,2179,2180,2183,2186,2188,2191,2193],{"class":240,"line":607},[292,2181,2182],{"class":304},"            f",[292,2184,2185],{"class":378},"\"CREATE TABLE IF NOT EXISTS ",[292,2187,847],{"class":365},[292,2189,2190],{"class":308},"name",[292,2192,853],{"class":365},[292,2194,2195],{"class":378}," PARTITION OF usage_event \"\n",[292,2197,2198,2200,2203,2205,2208,2210,2213,2215,2218,2220],{"class":240,"line":613},[292,2199,2182],{"class":304},[292,2201,2202],{"class":378},"\"FOR VALUES FROM ('",[292,2204,847],{"class":365},[292,2206,2207],{"class":308},"start.isoformat()",[292,2209,853],{"class":365},[292,2211,2212],{"class":378},"') TO ('",[292,2214,847],{"class":365},[292,2216,2217],{"class":308},"end.isoformat()",[292,2219,853],{"class":365},[292,2221,2222],{"class":378},"')\"\n",[292,2224,2225],{"class":240,"line":624},[292,2226,2227],{"class":308},"        )\n",[292,2229,2230],{"class":240,"line":641},[292,2231,2232],{"class":308},"        created.append(name)\n",[292,2234,2235,2237],{"class":240,"line":658},[292,2236,761],{"class":304},[292,2238,2239],{"class":308}," created\n",[292,2241,2242],{"class":240,"line":679},[292,2243,346],{"emptyLinePlaceholder":345},[292,2245,2246],{"class":240,"line":696},[292,2247,346],{"emptyLinePlaceholder":345},[292,2249,2250,2252,2254,2257,2259,2261],{"class":240,"line":713},[292,2251,589],{"class":304},[292,2253,592],{"class":304},[292,2255,2256],{"class":595}," drop_expired",[292,2258,2070],{"class":308},[292,2260,926],{"class":365},[292,2262,604],{"class":308},[292,2264,2265,2268,2270,2273,2275,2278,2281,2283,2285],{"class":240,"line":719},[292,2266,2267],{"class":308},"    cutoff ",[292,2269,560],{"class":304},[292,2271,2272],{"class":308}," _month_start(today ",[292,2274,690],{"class":304},[292,2276,2277],{"class":308}," timedelta(",[292,2279,2280],{"class":566},"days",[292,2282,560],{"class":304},[292,2284,1942],{"class":365},[292,2286,388],{"class":308},[292,2288,2289,2292,2294,2296],{"class":240,"line":727},[292,2290,2291],{"class":308},"    rows ",[292,2293,560],{"class":304},[292,2295,988],{"class":304},[292,2297,2298],{"class":308}," conn.fetch(\n",[292,2300,2301],{"class":240,"line":738},[292,2302,2303],{"class":378},"        \"SELECT c.relname FROM pg_class c \"\n",[292,2305,2306],{"class":240,"line":753},[292,2307,2308],{"class":378},"        \"JOIN pg_inherits i ON i.inhrelid = c.oid \"\n",[292,2310,2311],{"class":240,"line":758},[292,2312,2313],{"class":378},"        \"JOIN pg_class p ON p.oid = i.inhparent \"\n",[292,2315,2316],{"class":240,"line":767},[292,2317,2318],{"class":378},"        \"WHERE p.relname = 'usage_event' ORDER BY c.relname\"\n",[292,2320,2321],{"class":240,"line":772},[292,2322,2323],{"class":308},"    )\n",[292,2325,2326,2329,2331],{"class":240,"line":777},[292,2327,2328],{"class":308},"    dropped ",[292,2330,560],{"class":304},[292,2332,2084],{"class":308},[292,2334,2335,2337,2339,2341],{"class":240,"line":799},[292,2336,2089],{"class":304},[292,2338,906],{"class":308},[292,2340,909],{"class":304},[292,2342,2343],{"class":308}," rows:\n",[292,2345,2346,2348,2350,2353,2356],{"class":240,"line":827},[292,2347,2143],{"class":308},[292,2349,560],{"class":304},[292,2351,2352],{"class":308}," row[",[292,2354,2355],{"class":378},"\"relname\"",[292,2357,540],{"class":308},[292,2359,2360,2363,2365,2368,2371],{"class":240,"line":869},[292,2361,2362],{"class":308},"        stamp ",[292,2364,560],{"class":304},[292,2366,2367],{"class":308}," name.removeprefix(",[292,2369,2370],{"class":378},"\"usage_event_\"",[292,2372,476],{"class":308},[292,2374,2375,2378,2380,2383,2385,2388,2391,2394,2396,2399,2402],{"class":240,"line":900},[292,2376,2377],{"class":308},"        year, month ",[292,2379,560],{"class":304},[292,2381,2382],{"class":308}," (",[292,2384,1979],{"class":365},[292,2386,2387],{"class":308},"(part) ",[292,2389,2390],{"class":304},"for",[292,2392,2393],{"class":308}," part ",[292,2395,909],{"class":304},[292,2397,2398],{"class":308}," stamp.split(",[292,2400,2401],{"class":378},"\"_\"",[292,2403,388],{"class":308},[292,2405,2406,2408,2411,2413,2415,2417],{"class":240,"line":915},[292,2407,699],{"class":304},[292,2409,2410],{"class":308}," date(year, month, ",[292,2412,1270],{"class":365},[292,2414,891],{"class":308},[292,2416,670],{"class":304},[292,2418,2419],{"class":308}," cutoff:\n",[292,2421,2422,2424,2426,2429,2432,2434,2436,2438,2440],{"class":240,"line":943},[292,2423,1083],{"class":304},[292,2425,1689],{"class":308},[292,2427,2428],{"class":304},"f",[292,2430,2431],{"class":378},"\"ALTER TABLE usage_event DETACH PARTITION ",[292,2433,847],{"class":365},[292,2435,2190],{"class":308},[292,2437,853],{"class":365},[292,2439,844],{"class":378},[292,2441,476],{"class":308},[292,2443,2444,2446,2448,2450,2453,2455,2457,2459,2461],{"class":240,"line":948},[292,2445,1083],{"class":304},[292,2447,1689],{"class":308},[292,2449,2428],{"class":304},[292,2451,2452],{"class":378},"\"DROP TABLE ",[292,2454,847],{"class":365},[292,2456,2190],{"class":308},[292,2458,853],{"class":365},[292,2460,844],{"class":378},[292,2462,476],{"class":308},[292,2464,2465],{"class":240,"line":953},[292,2466,2467],{"class":308},"            dropped.append(name)\n",[292,2469,2470,2472],{"class":240,"line":970},[292,2471,761],{"class":304},[292,2473,2474],{"class":308}," dropped\n",[292,2476,2477],{"class":240,"line":980},[292,2478,346],{"emptyLinePlaceholder":345},[292,2480,2481],{"class":240,"line":994},[292,2482,346],{"emptyLinePlaceholder":345},[292,2484,2485,2487,2489,2491,2493,2495],{"class":240,"line":1001},[292,2486,589],{"class":304},[292,2488,592],{"class":304},[292,2490,1649],{"class":595},[292,2492,1652],{"class":308},[292,2494,794],{"class":365},[292,2496,676],{"class":308},[292,2498,2499,2501,2503,2505,2507,2509,2511,2513],{"class":240,"line":1018},[292,2500,1661],{"class":308},[292,2502,560],{"class":304},[292,2504,988],{"class":304},[292,2506,1668],{"class":308},[292,2508,468],{"class":378},[292,2510,382],{"class":308},[292,2512,473],{"class":378},[292,2514,388],{"class":308},[292,2516,2517,2519],{"class":240,"line":1027},[292,2518,1297],{"class":304},[292,2520,676],{"class":308},[292,2522,2523,2526,2528],{"class":240,"line":1051},[292,2524,2525],{"class":308},"        today ",[292,2527,560],{"class":304},[292,2529,2530],{"class":308}," date.today()\n",[292,2532,2533,2536,2539,2542,2545,2547],{"class":240,"line":1057},[292,2534,2535],{"class":365},"        print",[292,2537,2538],{"class":308},"({",[292,2540,2541],{"class":378},"\"created\"",[292,2543,2544],{"class":308},": ",[292,2546,635],{"class":304},[292,2548,2549],{"class":308}," ensure_partitions(conn, today),\n",[292,2551,2552,2555,2557,2559],{"class":240,"line":1071},[292,2553,2554],{"class":378},"               \"dropped\"",[292,2556,2544],{"class":308},[292,2558,635],{"class":304},[292,2560,2561],{"class":308}," drop_expired(conn, today)})\n",[292,2563,2564,2566],{"class":240,"line":1080},[292,2565,1309],{"class":304},[292,2567,676],{"class":308},[292,2569,2570,2572],{"class":240,"line":1094},[292,2571,1351],{"class":304},[292,2573,1706],{"class":308},[292,2575,2576],{"class":240,"line":1102},[292,2577,346],{"emptyLinePlaceholder":345},[292,2579,2580],{"class":240,"line":1115},[292,2581,346],{"emptyLinePlaceholder":345},[292,2583,2585,2587,2589,2591,2593],{"class":240,"line":2584},61,[292,2586,1719],{"class":304},[292,2588,1722],{"class":365},[292,2590,1725],{"class":304},[292,2592,1728],{"class":378},[292,2594,676],{"class":308},[292,2596,2598],{"class":240,"line":2597},62,[292,2599,1735],{"class":308},[14,2601,2602,2603,2606,2607,2610,2611,2614,2615,2619],{},"Detach before dropping: ",[24,2604,2605],{},"DETACH PARTITION"," takes a brief lock and removes the child from the parent's plan, then ",[24,2608,2609],{},"DROP TABLE"," frees the files without blocking the parent. Roll the day's events into ",[24,2612,2613],{},"usage_rollup_daily"," before retention runs, and your ",[18,2616,2618],{"href":2617},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Ftracking-api-usage-and-analytics\u002Fbuilding-a-customer-usage-dashboard\u002F","customer usage dashboard"," keeps working on history that no longer exists in raw form.",[137,2621,146,2625,146,2628,146,2631,146,2633,146,2636,146,2642,146,2645,146,2650,146,2653,146,2657,146,2659,146,2662,146,2666,146,2669,146,2672,146,2676,146,2678,146,2681,146,2685,146,2688,146,2691,146,2695,146,2698,146,2700,146,2704,146,2706,146,2709,146,2713,146,2718],{"viewBox":139,"role":140,"ariaLabelledBy":2622,"xmlns":144,"style":145},[2623,2624],"luep-part-t","luep-part-d",[148,2626,2627],{"id":2623},"Monthly partition lifecycle and retention",[152,2629,2630],{"id":2624},"Seven monthly partitions in a row: the three oldest are detached and dropped past the ninety day cutoff, the current month is live, two future months are pre-created, and a daily rollup table spans all of them.",[156,2632],{"x":158,"y":158,"width":159,"height":160,"fill":161},[163,2634,2635],{"x":165,"y":180,"fill":167,"style":168},"usage_event partitions",[156,2637],{"x":165,"y":2638,"width":2639,"height":2640,"rx":2641,"fill":161,"stroke":191,"style":1797},"48","88","56","6",[163,2643,2644],{"x":1771,"y":1795,"fill":167,"style":1407},"2026_02",[163,2646,2649],{"x":1771,"y":2647,"fill":222,"style":2648},"90","text-anchor:middle;font-size:10;font-family:var(--font-sans);","dropped",[156,2651],{"x":2652,"y":2638,"width":2639,"height":2640,"rx":2641,"fill":161,"stroke":191,"style":1797},"116",[163,2654,2656],{"x":2655,"y":1795,"fill":167,"style":1407},"160","2026_03",[163,2658,2649],{"x":2655,"y":2647,"fill":222,"style":2648},[156,2660],{"x":2661,"y":2638,"width":2639,"height":2640,"rx":2641,"fill":1773,"stroke":244,"style":1377},"212",[163,2663,2665],{"x":2664,"y":1795,"fill":167,"style":1407},"256","2026_05",[163,2667,2668],{"x":2664,"y":2647,"fill":222,"style":2648},"retained",[156,2670],{"x":2671,"y":2638,"width":2639,"height":2640,"rx":2641,"fill":1773,"stroke":244,"style":1377},"308",[163,2673,2675],{"x":2674,"y":1795,"fill":167,"style":1407},"352","2026_06",[163,2677,2668],{"x":2674,"y":2647,"fill":222,"style":2648},[156,2679],{"x":2680,"y":2638,"width":2639,"height":2640,"rx":2641,"fill":161,"stroke":186,"style":1392},"404",[163,2682,2684],{"x":2683,"y":1795,"fill":167,"style":1407},"448","2026_07",[163,2686,2687],{"x":2683,"y":2647,"fill":222,"style":2648},"writing now",[156,2689],{"x":1385,"y":2638,"width":2639,"height":2640,"rx":2641,"fill":161,"stroke":181,"style":2690},"stroke-width:1.5;stroke-dasharray:5 4;",[163,2692,2694],{"x":2693,"y":1795,"fill":167,"style":1407},"544","2026_08",[163,2696,2697],{"x":2693,"y":2647,"fill":222,"style":2648},"pre-created",[156,2699],{"x":189,"y":2638,"width":2639,"height":2640,"rx":2641,"fill":161,"stroke":181,"style":2690},[163,2701,2703],{"x":2702,"y":1795,"fill":167,"style":1407},"640","2026_09",[163,2705,2697],{"x":2702,"y":2647,"fill":222,"style":2648},[240,2707],{"x1":2708,"y1":203,"x2":2708,"y2":1832,"stroke":191,"style":1797},"204",[163,2710,2712],{"x":2708,"y":2711,"fill":167,"style":1407},"162","90-day cutoff: DETACH then DROP",[156,2714],{"x":165,"y":2715,"width":2716,"height":1769,"rx":2641,"fill":161,"stroke":2717,"style":1797},"184","664","var(--c-yellow)",[163,2719,2721],{"x":2674,"y":2720,"fill":167,"style":1776},"209","usage_rollup_daily — one row per day, customer, route — kept 25 months",[32,2723,2725],{"id":2724},"reconciling-the-log-against-stripe","Reconciling the log against Stripe",[14,2727,2728],{},"Your Postgres log and Stripe's meter are independent counters, and independent counters drift. Events get spooled and never replayed, a deploy kills a pod mid-flush, a retry double-reports. Run reconciliation the day before each invoice finalizes: aggregate your rollup for the period, pull Stripe's aggregate for the same window, alert on any gap over half a percent. That threshold is deliberate — chasing single-event differences wastes a week, while a systematic gap shows up immediately at that resolution.",[283,2730,2732],{"className":285,"code":2731,"language":287,"meta":288,"style":288},"# usage\u002Freconcile.py\nimport asyncio\nimport os\n\nimport asyncpg\nimport stripe\n\nstripe.api_key = os.getenv(\"STRIPE_SECRET_KEY\")\nDRIFT_ALERT_RATIO = float(os.getenv(\"USAGE_DRIFT_ALERT_RATIO\", \"0.005\"))\n\nLOCAL_TOTAL_SQL = \"\"\"\nSELECT COALESCE(SUM(billable_units), 0)::bigint AS units\nFROM usage_rollup_daily\nWHERE customer_id = $1 AND day >= $2 AND day \u003C $3\n\"\"\"\n\n\nasync def local_units(conn: asyncpg.Connection, customer_id: str, start, end) -> int:\n    return await conn.fetchval(LOCAL_TOTAL_SQL, customer_id, start, end)\n\n\ndef stripe_units(meter_id: str, stripe_customer_id: str, start_ts: int, end_ts: int) -> int:\n    summaries = stripe.billing.Meter.list_event_summaries(\n        meter_id, customer=stripe_customer_id, start_time=start_ts, end_time=end_ts\n    )\n    return int(sum(float(s[\"aggregated_value\"]) for s in summaries.auto_paging_iter()))\n\n\nasync def check(customer_id: str, stripe_customer_id: str, start, end,\n                start_ts: int, end_ts: int) -> dict:\n    conn = await asyncpg.connect(os.getenv(\"USAGE_DATABASE_URL\", \"postgresql:\u002F\u002Flocalhost\u002Fapi\"))\n    try:\n        mine = await local_units(conn, customer_id, start, end)\n    finally:\n        await conn.close()\n    theirs = stripe_units(os.getenv(\"STRIPE_METER_ID\", \"\"), stripe_customer_id, start_ts, end_ts)\n    drift = theirs - mine\n    ratio = abs(drift) \u002F mine if mine else 0.0\n    match ratio:\n        case r if r == 0:\n            status = \"exact\"\n        case r if r \u003C DRIFT_ALERT_RATIO:\n            status = \"within_tolerance\"\n        case _:\n            status = \"investigate\"\n    return {\"local\": mine, \"stripe\": theirs, \"drift\": drift, \"status\": status}\n",[24,2733,2734,2739,2745,2751,2755,2761,2768,2772,2786,2807,2811,2820,2825,2830,2835,2839,2843,2847,2868,2882,2886,2890,2924,2934,2963,2967,3002,3006,3010,3031,3049,3067,3073,3085,3091,3097,3118,3133,3162,3170,3189,3199,3216,3225,3232,3241],{"__ignoreMap":288},[292,2735,2736],{"class":240,"line":294},[292,2737,2738],{"class":297},"# usage\u002Freconcile.py\n",[292,2740,2741,2743],{"class":240,"line":301},[292,2742,305],{"class":304},[292,2744,309],{"class":308},[292,2746,2747,2749],{"class":240,"line":312},[292,2748,305],{"class":304},[292,2750,325],{"class":308},[292,2752,2753],{"class":240,"line":320},[292,2754,346],{"emptyLinePlaceholder":345},[292,2756,2757,2759],{"class":240,"line":328},[292,2758,305],{"class":304},[292,2760,354],{"class":308},[292,2762,2763,2765],{"class":240,"line":342},[292,2764,305],{"class":304},[292,2766,2767],{"class":308}," stripe\n",[292,2769,2770],{"class":240,"line":349},[292,2771,346],{"emptyLinePlaceholder":345},[292,2773,2774,2777,2779,2781,2784],{"class":240,"line":357},[292,2775,2776],{"class":308},"stripe.api_key ",[292,2778,560],{"class":304},[292,2780,465],{"class":308},[292,2782,2783],{"class":378},"\"STRIPE_SECRET_KEY\"",[292,2785,476],{"class":308},[292,2787,2788,2791,2793,2795,2797,2800,2802,2805],{"class":240,"line":362},[292,2789,2790],{"class":365},"DRIFT_ALERT_RATIO",[292,2792,369],{"class":304},[292,2794,399],{"class":365},[292,2796,375],{"class":308},[292,2798,2799],{"class":378},"\"USAGE_DRIFT_ALERT_RATIO\"",[292,2801,382],{"class":308},[292,2803,2804],{"class":378},"\"0.005\"",[292,2806,388],{"class":308},[292,2808,2809],{"class":240,"line":391},[292,2810,346],{"emptyLinePlaceholder":345},[292,2812,2813,2816,2818],{"class":240,"line":414},[292,2814,2815],{"class":365},"LOCAL_TOTAL_SQL",[292,2817,369],{"class":304},[292,2819,1510],{"class":378},[292,2821,2822],{"class":240,"line":436},[292,2823,2824],{"class":378},"SELECT COALESCE(SUM(billable_units), 0)::bigint AS units\n",[292,2826,2827],{"class":240,"line":457},[292,2828,2829],{"class":378},"FROM usage_rollup_daily\n",[292,2831,2832],{"class":240,"line":479},[292,2833,2834],{"class":378},"WHERE customer_id = $1 AND day >= $2 AND day \u003C $3\n",[292,2836,2837],{"class":240,"line":484},[292,2838,1632],{"class":378},[292,2840,2841],{"class":240,"line":495},[292,2842,346],{"emptyLinePlaceholder":345},[292,2844,2845],{"class":240,"line":519},[292,2846,346],{"emptyLinePlaceholder":345},[292,2848,2849,2851,2853,2856,2859,2861,2864,2866],{"class":240,"line":537},[292,2850,589],{"class":304},[292,2852,592],{"class":304},[292,2854,2855],{"class":595}," local_units",[292,2857,2858],{"class":308},"(conn: asyncpg.Connection, customer_id: ",[292,2860,926],{"class":365},[292,2862,2863],{"class":308},", start, end) -> ",[292,2865,1979],{"class":365},[292,2867,676],{"class":308},[292,2869,2870,2872,2874,2877,2879],{"class":240,"line":543},[292,2871,761],{"class":304},[292,2873,988],{"class":304},[292,2875,2876],{"class":308}," conn.fetchval(",[292,2878,2815],{"class":365},[292,2880,2881],{"class":308},", customer_id, start, end)\n",[292,2883,2884],{"class":240,"line":548},[292,2885,346],{"emptyLinePlaceholder":345},[292,2887,2888],{"class":240,"line":576},[292,2889,346],{"emptyLinePlaceholder":345},[292,2891,2892,2894,2897,2900,2902,2905,2907,2910,2912,2915,2917,2920,2922],{"class":240,"line":581},[292,2893,780],{"class":304},[292,2895,2896],{"class":595}," stripe_units",[292,2898,2899],{"class":308},"(meter_id: ",[292,2901,926],{"class":365},[292,2903,2904],{"class":308},", stripe_customer_id: ",[292,2906,926],{"class":365},[292,2908,2909],{"class":308},", start_ts: ",[292,2911,1979],{"class":365},[292,2913,2914],{"class":308},", end_ts: ",[292,2916,1979],{"class":365},[292,2918,2919],{"class":308},") -> ",[292,2921,1979],{"class":365},[292,2923,676],{"class":308},[292,2925,2926,2929,2931],{"class":240,"line":586},[292,2927,2928],{"class":308},"    summaries ",[292,2930,560],{"class":304},[292,2932,2933],{"class":308}," stripe.billing.Meter.list_event_summaries(\n",[292,2935,2936,2939,2942,2944,2947,2950,2952,2955,2958,2960],{"class":240,"line":607},[292,2937,2938],{"class":308},"        meter_id, ",[292,2940,2941],{"class":566},"customer",[292,2943,560],{"class":304},[292,2945,2946],{"class":308},"stripe_customer_id, ",[292,2948,2949],{"class":566},"start_time",[292,2951,560],{"class":304},[292,2953,2954],{"class":308},"start_ts, ",[292,2956,2957],{"class":566},"end_time",[292,2959,560],{"class":304},[292,2961,2962],{"class":308},"end_ts\n",[292,2964,2965],{"class":240,"line":613},[292,2966,2323],{"class":308},[292,2968,2969,2971,2973,2975,2978,2980,2983,2986,2989,2992,2994,2997,2999],{"class":240,"line":624},[292,2970,761],{"class":304},[292,2972,372],{"class":365},[292,2974,2100],{"class":308},[292,2976,2977],{"class":365},"sum",[292,2979,2100],{"class":308},[292,2981,2982],{"class":365},"float",[292,2984,2985],{"class":308},"(s[",[292,2987,2988],{"class":378},"\"aggregated_value\"",[292,2990,2991],{"class":308},"]) ",[292,2993,2390],{"class":304},[292,2995,2996],{"class":308}," s ",[292,2998,909],{"class":304},[292,3000,3001],{"class":308}," summaries.auto_paging_iter()))\n",[292,3003,3004],{"class":240,"line":641},[292,3005,346],{"emptyLinePlaceholder":345},[292,3007,3008],{"class":240,"line":658},[292,3009,346],{"emptyLinePlaceholder":345},[292,3011,3012,3014,3016,3019,3022,3024,3026,3028],{"class":240,"line":679},[292,3013,589],{"class":304},[292,3015,592],{"class":304},[292,3017,3018],{"class":595}," check",[292,3020,3021],{"class":308},"(customer_id: ",[292,3023,926],{"class":365},[292,3025,2904],{"class":308},[292,3027,926],{"class":365},[292,3029,3030],{"class":308},", start, end,\n",[292,3032,3033,3036,3038,3040,3042,3044,3047],{"class":240,"line":696},[292,3034,3035],{"class":308},"                start_ts: ",[292,3037,1979],{"class":365},[292,3039,2914],{"class":308},[292,3041,1979],{"class":365},[292,3043,2919],{"class":308},[292,3045,3046],{"class":365},"dict",[292,3048,676],{"class":308},[292,3050,3051,3053,3055,3057,3059,3061,3063,3065],{"class":240,"line":713},[292,3052,1661],{"class":308},[292,3054,560],{"class":304},[292,3056,988],{"class":304},[292,3058,1668],{"class":308},[292,3060,468],{"class":378},[292,3062,382],{"class":308},[292,3064,473],{"class":378},[292,3066,388],{"class":308},[292,3068,3069,3071],{"class":240,"line":719},[292,3070,1297],{"class":304},[292,3072,676],{"class":308},[292,3074,3075,3078,3080,3082],{"class":240,"line":727},[292,3076,3077],{"class":308},"        mine ",[292,3079,560],{"class":304},[292,3081,988],{"class":304},[292,3083,3084],{"class":308}," local_units(conn, customer_id, start, end)\n",[292,3086,3087,3089],{"class":240,"line":738},[292,3088,1309],{"class":304},[292,3090,676],{"class":308},[292,3092,3093,3095],{"class":240,"line":753},[292,3094,1351],{"class":304},[292,3096,1706],{"class":308},[292,3098,3099,3102,3104,3107,3110,3112,3115],{"class":240,"line":758},[292,3100,3101],{"class":308},"    theirs ",[292,3103,560],{"class":304},[292,3105,3106],{"class":308}," stripe_units(os.getenv(",[292,3108,3109],{"class":378},"\"STRIPE_METER_ID\"",[292,3111,382],{"class":308},[292,3113,3114],{"class":378},"\"\"",[292,3116,3117],{"class":308},"), stripe_customer_id, start_ts, end_ts)\n",[292,3119,3120,3123,3125,3128,3130],{"class":240,"line":767},[292,3121,3122],{"class":308},"    drift ",[292,3124,560],{"class":304},[292,3126,3127],{"class":308}," theirs ",[292,3129,690],{"class":304},[292,3131,3132],{"class":308}," mine\n",[292,3134,3135,3138,3140,3143,3146,3149,3152,3154,3156,3159],{"class":240,"line":772},[292,3136,3137],{"class":308},"    ratio ",[292,3139,560],{"class":304},[292,3141,3142],{"class":365}," abs",[292,3144,3145],{"class":308},"(drift) ",[292,3147,3148],{"class":304},"\u002F",[292,3150,3151],{"class":308}," mine ",[292,3153,1719],{"class":304},[292,3155,3151],{"class":308},[292,3157,3158],{"class":304},"else",[292,3160,3161],{"class":365}," 0.0\n",[292,3163,3164,3167],{"class":240,"line":777},[292,3165,3166],{"class":304},"    match",[292,3168,3169],{"class":308}," ratio:\n",[292,3171,3172,3175,3178,3180,3182,3185,3187],{"class":240,"line":799},[292,3173,3174],{"class":304},"        case",[292,3176,3177],{"class":308}," r ",[292,3179,1719],{"class":304},[292,3181,3177],{"class":308},[292,3183,3184],{"class":304},"==",[292,3186,708],{"class":365},[292,3188,676],{"class":308},[292,3190,3191,3194,3196],{"class":240,"line":827},[292,3192,3193],{"class":308},"            status ",[292,3195,560],{"class":304},[292,3197,3198],{"class":378}," \"exact\"\n",[292,3200,3201,3203,3205,3207,3209,3211,3214],{"class":240,"line":869},[292,3202,3174],{"class":304},[292,3204,3177],{"class":308},[292,3206,1719],{"class":304},[292,3208,3177],{"class":308},[292,3210,670],{"class":304},[292,3212,3213],{"class":365}," DRIFT_ALERT_RATIO",[292,3215,676],{"class":308},[292,3217,3218,3220,3222],{"class":240,"line":900},[292,3219,3193],{"class":308},[292,3221,560],{"class":304},[292,3223,3224],{"class":378}," \"within_tolerance\"\n",[292,3226,3227,3229],{"class":240,"line":915},[292,3228,3174],{"class":304},[292,3230,3231],{"class":308}," _:\n",[292,3233,3234,3236,3238],{"class":240,"line":943},[292,3235,3193],{"class":308},[292,3237,560],{"class":304},[292,3239,3240],{"class":378}," \"investigate\"\n",[292,3242,3243,3245,3248,3251,3254,3257,3260,3263,3266,3269],{"class":240,"line":948},[292,3244,761],{"class":304},[292,3246,3247],{"class":308}," {",[292,3249,3250],{"class":378},"\"local\"",[292,3252,3253],{"class":308},": mine, ",[292,3255,3256],{"class":378},"\"stripe\"",[292,3258,3259],{"class":308},": theirs, ",[292,3261,3262],{"class":378},"\"drift\"",[292,3264,3265],{"class":308},": drift, ",[292,3267,3268],{"class":378},"\"status\"",[292,3270,3271],{"class":308},": status}\n",[14,3273,3274,3275,3279,3280,3284],{},"Postgres is the authority here, not Stripe. When they disagree, you replay missing meter events from your rollup rather than adjusting your log to match the invoice — the details of that reporting path live in the ",[18,3276,3278],{"href":3277},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fstripe-metered-billing-configuration\u002F","Stripe metered billing configuration"," page. Log every reconciliation result as a structured record so a support conversation six months later has evidence; the ",[18,3281,3283],{"href":3282},"\u002Fscaling-and-operating-production-python-apis\u002Fmonitoring-and-logging-python-apis\u002Fstructured-logging-with-structlog\u002F","structured logging setup"," makes those searchable without extra work.",[137,3286,146,3291,146,3294,146,3297,146,3300,146,3304,146,3308,146,3312,146,3314,146,3317,146,3321,146,3326,146,3330,146,3333,146,3338,146,3341,146,3346,146,3350,146,3354,146,3358,146,3361,146,3365,146,3368,146,3370,146,3374,146,3378],{"viewBox":3287,"role":140,"ariaLabelledBy":3288,"xmlns":144,"style":145},"0 0 720 270",[3289,3290],"luep-rec-t","luep-rec-d",[148,3292,3293],{"id":3289},"Reconciliation decision path before an invoice finalizes",[152,3295,3296],{"id":3290},"The rollup total is compared against the Stripe meter aggregate; an exact or near match closes the period, while drift above half a percent triggers replaying missing meter events from Postgres.",[156,3298],{"x":158,"y":158,"width":159,"height":3299,"fill":161},"270",[156,3301],{"x":165,"y":180,"width":3302,"height":3303,"rx":1772,"fill":161,"stroke":186,"style":1797},"170","58",[163,3305,2613],{"x":3306,"y":3307,"fill":167,"style":1776},"105","54",[163,3309,3311],{"x":3306,"y":3310,"fill":222,"style":1407},"73","SUM(billable_units)",[156,3313],{"x":165,"y":212,"width":3302,"height":3303,"rx":1772,"fill":161,"stroke":2717,"style":1797},[163,3315,3316],{"x":3306,"y":184,"fill":167,"style":1776},"Stripe meter",[163,3318,3320],{"x":3306,"y":3319,"fill":222,"style":1407},"183","event summaries",[240,3322],{"x1":1784,"y1":3323,"x2":3324,"y2":3325,"stroke":222,"style":1797},"59","248","98",[240,3327],{"x1":1784,"y1":3328,"x2":3324,"y2":3329,"stroke":222,"style":1797},"169","130",[156,3331],{"x":160,"y":3332,"width":2655,"height":178,"rx":1772,"fill":1773,"stroke":244,"style":1377},"84",[163,3334,3337],{"x":3335,"y":3336,"fill":167,"style":1776},"330","109","compare ratio",[163,3339,3340],{"x":3335,"y":1840,"fill":222,"style":1407},"|drift| \u002F local",[240,3342],{"x1":3343,"y1":3344,"x2":3345,"y2":1383,"stroke":181,"style":1797},"410","102","472",[240,3347],{"x1":3343,"y1":3348,"x2":3345,"y2":3349,"stroke":191,"style":1797},"126","176",[163,3351,3353],{"x":3352,"y":1795,"fill":222,"style":2648},"440","\u003C 0.5%",[163,3355,3357],{"x":3356,"y":2655,"fill":222,"style":2648},"443",">= 0.5%",[156,3359],{"x":3360,"y":180,"width":1396,"height":3303,"rx":1772,"fill":161,"stroke":181,"style":1797},"474",[163,3362,3364],{"x":3363,"y":3307,"fill":167,"style":1776},"582","close the period",[163,3366,3367],{"x":3363,"y":3310,"fill":222,"style":1407},"let the invoice finalize",[156,3369],{"x":3360,"y":177,"width":1396,"height":3303,"rx":1772,"fill":161,"stroke":191,"style":1797},[163,3371,3373],{"x":3363,"y":3372,"fill":167,"style":1776},"174","replay meter events",[163,3375,3377],{"x":3363,"y":3376,"fill":222,"style":1407},"193","Postgres is the authority",[163,3379,3381],{"x":165,"y":3380,"fill":222,"style":223},"242","Run one day before the billing period closes, while corrections are still free.",[32,3383,3385],{"id":3384},"when-to-choose-this-and-when-to-avoid-it","When to choose this, and when to avoid it",[14,3387,3388,3389,3393],{},"Choose the buffered Postgres log when you run under roughly 2,000 requests per second per instance, usage drives billing, and you want usage joinable against customers without a second data store. That covers virtually every API business earning under a million dollars a year. At 500 requests per second you write about 1.3 billion rows a year, but 90-day retention keeps the live table near 320 million rows and 35 GB — comfortable on a managed instance costing $50 to $90 a month, a fraction of a cent per thousand requests once you work through the ",[18,3390,3392],{"href":3391},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fcalculating-cost-per-api-request\u002F","cost per API request"," math.",[14,3395,3396],{},"Avoid it in three cases. If a single lost event is legally unacceptable — payments, regulated metering — put a durable buffer in front, either a Redis Stream or a queue, and accept the extra hop. If your analytics questions are open-ended slice-and-dice across billions of rows, a column store earns its keep and Postgres does not. And if you are running serverless functions with no long-lived process, the in-memory buffer has nowhere to live; write to a managed queue instead, because a function that exits after each request cannot flush anything.",[32,3398,3400],{"id":3399},"migrating-from-inline-inserts","Migrating from inline inserts",[14,3402,3403,3404,3406,3407,3410,3411,3414,3415,3418],{},"Switching from a synchronous insert to the buffered writer takes an afternoon and no downtime. First, deploy the partitioned ",[24,3405,1833],{}," table alongside the existing one and run ",[24,3408,3409],{},"ensure_partitions"," so the current and next two months exist. Second, ship the writer and lifespan hook with the middleware still inserting inline — a drain loop with nothing to drain proves startup and shutdown work. Third, flip the middleware to ",[24,3412,3413],{},"queue.put_nowait"," behind an environment flag on one instance and compare row counts across both tables for an hour. Fourth, backfill with one ",[24,3416,3417],{},"INSERT INTO usage_event SELECT ... FROM old_usage"," per month so each statement lands in a single partition. Finally, drop the old table and watch pool wait time fall.",[32,3420,3422],{"id":3421},"builder-verdict","Builder verdict",[14,3424,3425,3426,3428],{},"Buffer in process, write with ",[24,3427,26],{},", partition by month, and let Postgres be the authority your invoices are built on. This is the highest return-per-hour infrastructure in a monetized API: two files of code and a nightly maintenance job buy you billing you can defend, per-customer margin you can query, and a hot path measurably faster than the version that inserted a row per request. Every alternative costs more — a dedicated analytics store adds a bill and a sync problem before you have customers to justify it, and a log-shipping pipeline strands your revenue data where you cannot join it. Ship the buffered writer this week, add reconciliation before your first invoice cycle closes, and revisit the decision only when one Postgres instance genuinely stops keeping up.",[32,3430,3432],{"id":3431},"faq","FAQ",[14,3434,3435,3439,3440,3442],{},[3436,3437,3438],"strong",{},"How much does this cost to run at 50 million requests a month?","\nRoughly $50 to $90 a month of managed Postgres. Fifty million events at about 110 bytes plus index overhead is around 8 GB per month, and 90-day retention caps the live table near 25 GB. Compute barely moves: batched ",[24,3441,26],{}," at 500 rows per transaction is a rounding error next to serving the requests.",[14,3444,3445,3448],{},[3436,3446,3447],{},"What happens to billing if a pod dies mid-flush?","\nYou lose at most one flush interval of events, so two seconds at the default setting — a few hundred rows worth cents. Planned deploys lose nothing, because the shutdown hook drains the queue before the process exits. If even that is unacceptable, put a Redis Stream between the middleware and the writer and accept roughly 0.4 ms per request.",[14,3450,3451,3454],{},[3436,3452,3453],{},"Do I need TimescaleDB or ClickHouse instead?","\nNot until plain Postgres stops keeping up, which for most API businesses means well past a thousand requests per second sustained. Declarative partitioning plus a BRIN index covers the access pattern, and one database keeps usage joinable against customers. Migrating later is a data-copy problem, not a rewrite.",[14,3456,3457,3460],{},[3436,3458,3459],{},"How risky is migrating from inline inserts to buffered writes?","\nLow, if you stage it. Deploy the new table and drain loop first with the middleware unchanged, then flip writes behind an environment flag on one instance and compare counts for an hour before rolling out. The old path stays functional the whole time, so a rollback is one variable change, not a redeploy.",[14,3462,3463,3466],{},[3436,3464,3465],{},"Should the usage log live in the same database as my application data?","\nStart there, and split only when write volume starts affecting product query latency. A separate database means a second connection pool, a second backup policy, and joins you can no longer run. When you do split, the writer needs only a new DSN.",[32,3468,3470],{"id":3469},"related","Related",[3472,3473,3474,3480,3486,3492,3498],"ul",{},[3475,3476,3477,3479],"li",{},[18,3478,21],{"href":20}," — the full capture pipeline this table sits inside, including middleware and quota counters.",[3475,3481,3482,3485],{},[18,3483,3484],{"href":2617},"Building a Customer Usage Dashboard"," — turn these rollups into the screen customers check before they renew.",[3475,3487,3488,3491],{},[18,3489,3490],{"href":268},"asyncpg vs psycopg3 for FastAPI"," — driver choice matters most for the COPY path described here.",[3475,3493,3494,3497],{},[18,3495,3496],{"href":45},"Fixing Connection Pool Exhaustion"," — the failure mode inline usage inserts cause first.",[3475,3499,3500,3503],{},[18,3501,3502],{"href":3277},"Stripe Metered Billing Configuration"," — how the reconciled totals become invoice lines.",[3505,3506,3507],"style",{},"html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":288,"searchDepth":301,"depth":301,"links":3509},[3510,3511,3512,3513,3514,3515,3516,3517,3518,3519],{"id":34,"depth":301,"text":35},{"id":255,"depth":301,"text":256},{"id":1434,"depth":301,"text":1435},{"id":1848,"depth":301,"text":1849},{"id":2724,"depth":301,"text":2725},{"id":3384,"depth":301,"text":3385},{"id":3399,"depth":301,"text":3400},{"id":3421,"depth":301,"text":3422},{"id":3431,"depth":301,"text":3432},{"id":3469,"depth":301,"text":3470},"Write durable API usage events to Postgres without touching request latency: buffered COPY writes, per-key indexes, monthly partitions, retention and Stripe reconciliation.","md",{"pageTitle":3523,"type":3524,"datePublished":3525,"dateModified":3525},"Logging API Usage Events to Postgres Without Latency","article","2026-07-23","\u002Fbuilding-monetizing-api-driven-micro-saas\u002Ftracking-api-usage-and-analytics\u002Flogging-api-usage-events-to-postgres",{"title":5,"description":3520},"building-monetizing-api-driven-micro-saas\u002Ftracking-api-usage-and-analytics\u002Flogging-api-usage-events-to-postgres\u002Findex","CSNFnWLAV0IoHmtYrvu9fAM2K0pb_fpVT2ELdyjdkto",{"@context":3531,"@type":3532,"mainEntity":3533},"https:\u002F\u002Fschema.org","FAQPage",[3534,3539,3542,3545,3548],{"@type":3535,"name":3438,"acceptedAnswer":3536},"Question",{"@type":3537,"text":3538},"Answer","Roughly $50 to $90 a month of managed Postgres. Fifty million events at about 110 bytes plus index overhead is around 8 GB per month, and 90-day retention caps the live table near 25 GB. Compute barely moves: batched COPY at 500 rows per transaction is a rounding error next to serving the requests.",{"@type":3535,"name":3447,"acceptedAnswer":3540},{"@type":3537,"text":3541},"You lose at most one flush interval of events, so two seconds at the default setting — a few hundred rows worth cents. Planned deploys lose nothing, because the shutdown hook drains the queue before the process exits. If even that is unacceptable, put a Redis Stream between the middleware and the writer and accept roughly 0.4 ms per request.",{"@type":3535,"name":3453,"acceptedAnswer":3543},{"@type":3537,"text":3544},"Not until plain Postgres stops keeping up, which for most API businesses means well past a thousand requests per second sustained. Declarative partitioning plus a BRIN index covers the access pattern, and one database keeps usage joinable against customers. Migrating later is a data-copy problem, not a rewrite.",{"@type":3535,"name":3459,"acceptedAnswer":3546},{"@type":3537,"text":3547},"Low, if you stage it. Deploy the new table and drain loop first with the middleware unchanged, then flip writes behind an environment flag on one instance and compare counts for an hour before rolling out. The old path stays functional the whole time, so a rollback is one variable change, not a redeploy.",{"@type":3535,"name":3465,"acceptedAnswer":3549},{"@type":3537,"text":3550},"Start there, and split only when write volume starts affecting product query latency. A separate database means a second connection pool, a second backup policy, and joins you can no longer run. When you do split, the writer needs only a new DSN.",1784887028388]