[{"data":1,"prerenderedAt":2886},["ShallowReactive",2],{"page-\u002Fautomating-side-hustle-operations-with-apis\u002Fscheduling-data-pipelines-with-cron\u002F":3,"faq-schema-\u002Fautomating-side-hustle-operations-with-apis\u002Fscheduling-data-pipelines-with-cron\u002F":2865},{"id":4,"title":5,"body":6,"description":2855,"extension":2856,"meta":2857,"navigation":339,"path":2861,"seo":2862,"stem":2863,"__hash__":2864},"content\u002Fautomating-side-hustle-operations-with-apis\u002Fscheduling-data-pipelines-with-cron\u002Findex.md","Scheduling Recurring Data Pipelines for a Side Hustle",{"type":7,"value":8,"toc":2839},"minimark",[9,13,23,31,162,165,170,183,186,223,226,280,291,293,297,304,738,754,769,771,775,778,908,916,918,922,929,1237,1264,1280,1282,1286,1293,1525,1528,1538,1555,1617,1619,1623,1633,1881,1897,1899,1903,1906,1918,2099,2114,2116,2120,2123,2314,2344,2346,2350,2464,2466,2470,2532,2534,2538,2541,2583,2586,2618,2632,2634,2638,2641,2715,2723,2725,2729,2735,2741,2753,2765,2774,2776,2780,2785,2810,2815,2835],[10,11,5],"h1",{"id":12},"scheduling-recurring-data-pipelines-for-a-side-hustle",[14,15,16,17,22],"p",{},"Your side hustle runs on overnight work: a nightly Shopify sync, a Monday-morning report rollup, a scraper that refreshes prices every six hours. Doing that by hand burns the margin you started the business for. This guide builds one idempotent pipeline function and then shows you the full spectrum of ways to run it on a schedule — OS cron with a Python entrypoint, APScheduler in-process, Celery beat for distributed workers, and managed serverless cron like Render cron and GitHub Actions — so you can pick the cheapest reliable option for your stage. Part of the ",[18,19,21],"a",{"href":20},"\u002Fautomating-side-hustle-operations-with-apis\u002F","Automating Side-Hustle Operations with APIs"," guide.",[14,24,25,26,30],{},"The hard part of scheduling is not \"run this every night.\" It is running it ",[27,28,29],"em",{},"exactly once",", on the right clock, surviving restarts, and screaming when it fails. Get any one of those wrong and you either corrupt data, double-charge a customer, or discover three weeks later that your \"nightly\" job died silently on day two. This guide solves all four, and it treats scheduling as a commercial decision — the wrong tool at your stage either wastes money on idle infrastructure or costs you an afternoon debugging a duplicate-order incident. We build the pipeline once, then wire the same function into four different clocks so you can swap schedulers as the business grows without rewriting the work itself.",[32,33,41,42,41,46,41,50,41,69,41,76,41,84,41,90,41,93,41,96,41,99,41,106,41,111,41,119,41,126,41,131,41,134,41,137,41,141,41,144,41,148,41,152,41,157],"svg",{"viewBox":34,"role":35,"ariaLabelledBy":36,"xmlns":39,"style":40},"0 0 720 250","img",[37,38],"cron-pipe-t","cron-pipe-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;height:auto;margin:1.5rem 0;font-family:var(--font-sans);","\n  ",[43,44,45],"title",{"id":37},"Scheduler triggering an extract-transform-load pipeline",[47,48,49],"desc",{"id":38},"A timeline of scheduled triggers fires a pipeline that runs extract, then transform, then load, writing a run-status log.",[51,52,53,54,41],"defs",{},"\n    ",[55,56,63,64,53],"marker",{"id":57,"viewBox":58,"refX":59,"refY":60,"markerWidth":61,"markerHeight":61,"orient":62},"cron-pipe-arrow","0 0 10 10","9","5","7","auto-start-reverse","\n      ",[65,66],"path",{"d":67,"fill":68},"M0 0 L10 5 L0 10 z","var(--c-text-muted)",[70,71],"rect",{"x":72,"y":72,"width":73,"height":74,"fill":75},"0","720","250","var(--c-surface)",[77,78],"line",{"x1":79,"y1":80,"x2":81,"y2":80,"stroke":82,"style":83},"30","46","690","var(--c-border)","stroke-width:2;",[85,86],"circle",{"cx":87,"cy":80,"r":88,"fill":89},"120","6","var(--c-yellow)",[85,91],{"cx":92,"cy":80,"r":88,"fill":89},"280",[85,94],{"cx":95,"cy":80,"r":88,"fill":89},"440",[85,97],{"cx":98,"cy":80,"r":88,"fill":89},"600",[100,101,105],"text",{"x":102,"y":103,"fill":68,"style":104},"360","26","text-anchor:middle;font-size:13;font-family:var(--font-sans);","Schedule: 0 2 * * * (nightly triggers)",[77,107],{"x1":92,"y1":108,"x2":92,"y2":109,"stroke":68,"style":110},"58","96","stroke-width:2;marker-end:url(#cron-pipe-arrow);",[70,112],{"x":113,"y":114,"width":115,"height":116,"rx":117,"fill":75,"stroke":118,"style":83},"40","110","180","60","10","var(--c-blue)",[100,120,125],{"x":121,"y":122,"fill":123,"style":124},"130","145","var(--c-text)","text-anchor:middle;font-size:14;font-family:var(--font-sans);","Extract",[77,127],{"x1":128,"y1":129,"x2":130,"y2":129,"stroke":68,"style":110},"220","140","270",[70,132],{"x":130,"y":114,"width":115,"height":116,"rx":117,"fill":75,"stroke":133,"style":83},"var(--c-teal)",[100,135,136],{"x":102,"y":122,"fill":123,"style":124},"Transform",[77,138],{"x1":139,"y1":129,"x2":140,"y2":129,"stroke":68,"style":110},"450","500",[70,142],{"x":140,"y":114,"width":115,"height":116,"rx":117,"fill":75,"stroke":143,"style":83},"var(--c-coral)",[100,145,147],{"x":146,"y":122,"fill":123,"style":124},"590","Load",[77,149],{"x1":102,"y1":150,"x2":102,"y2":151,"stroke":68,"style":110},"170","212",[70,153],{"x":154,"y":151,"width":154,"height":155,"rx":156,"fill":75,"stroke":82,"style":83},"240","28","8",[100,158,161],{"x":102,"y":159,"fill":68,"style":160},"230","text-anchor:middle;font-size:12;font-family:var(--font-sans);","structured run-status log",[163,164],"hr",{},[166,167,169],"h2",{"id":168},"prerequisites","Prerequisites",[14,171,172,173,177,178,182],{},"You need Python 3.11+ and a pipeline that does real work — pulling data from a vendor API, transforming it, and writing it somewhere durable. The patterns here build directly on the ",[18,174,176],{"href":175},"\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-social-media-posting\u002Fsync-shopify-orders-to-google-sheets-via-api\u002F","Shopify orders to Google Sheets sync","; think of that as the body of the function we are about to schedule. If your pipeline pulls from a source that has no API and you are scraping it, the same scheduling patterns apply — but read ",[18,179,181],{"href":180},"\u002Fautomating-side-hustle-operations-with-apis\u002Fweb-scraping-vs-official-apis\u002F","web scraping vs official APIs"," first, because a scraped source fails far more often than a documented API and your retry and alerting story has to be tighter.",[14,184,185],{},"Install the schedulers you intend to test:",[187,188,193],"pre",{"className":189,"code":190,"language":191,"meta":192,"style":192},"language-bash shiki shiki-themes github-light github-dark","pip install apscheduler celery \"redis>=5\" httpx python-dotenv\n","bash","",[194,195,196],"code",{"__ignoreMap":192},[197,198,200,204,208,211,214,217,220],"span",{"class":77,"line":199},1,[197,201,203],{"class":202},"sScJk","pip",[197,205,207],{"class":206},"sZZnC"," install",[197,209,210],{"class":206}," apscheduler",[197,212,213],{"class":206}," celery",[197,215,216],{"class":206}," \"redis>=5\"",[197,218,219],{"class":206}," httpx",[197,221,222],{"class":206}," python-dotenv\n",[14,224,225],{},"Set configuration in the environment — never in code:",[187,227,231],{"className":228,"code":229,"language":230,"meta":192,"style":192},"language-env shiki shiki-themes github-light github-dark","PIPELINE_NAME=nightly-orders-sync\nCRON_SCHEDULE=0 2 * * *\nPIPELINE_TZ=Europe\u002FBrussels\nLOCK_PATH=\u002Ftmp\u002Fnightly-orders-sync.lock\nLOCK_TTL_SECONDS=3600\nSOURCE_API_URL=https:\u002F\u002Fapi.example.com\u002Forders\nSOURCE_API_KEY=replace-me\nCELERY_BROKER_URL=redis:\u002F\u002Flocalhost:6379\u002F0\n","env",[194,232,233,238,244,250,256,262,268,274],{"__ignoreMap":192},[197,234,235],{"class":77,"line":199},[197,236,237],{},"PIPELINE_NAME=nightly-orders-sync\n",[197,239,241],{"class":77,"line":240},2,[197,242,243],{},"CRON_SCHEDULE=0 2 * * *\n",[197,245,247],{"class":77,"line":246},3,[197,248,249],{},"PIPELINE_TZ=Europe\u002FBrussels\n",[197,251,253],{"class":77,"line":252},4,[197,254,255],{},"LOCK_PATH=\u002Ftmp\u002Fnightly-orders-sync.lock\n",[197,257,259],{"class":77,"line":258},5,[197,260,261],{},"LOCK_TTL_SECONDS=3600\n",[197,263,265],{"class":77,"line":264},6,[197,266,267],{},"SOURCE_API_URL=https:\u002F\u002Fapi.example.com\u002Forders\n",[197,269,271],{"class":77,"line":270},7,[197,272,273],{},"SOURCE_API_KEY=replace-me\n",[197,275,277],{"class":77,"line":276},8,[197,278,279],{},"CELERY_BROKER_URL=redis:\u002F\u002Flocalhost:6379\u002F0\n",[14,281,282,283,286,287,290],{},"Versions matter here more than usual. APScheduler 3.x and 4.x ship genuinely different APIs — 4.x moved to a data-store-backed engine and renamed the scheduler classes — so a tutorial written for one silently fails on the other. This guide uses 3.x, which is what ships stable today and what you will find in most production side-hustle stacks. Celery beat needs a broker such as Redis; without one the beat process starts but never delivers a task. Pin all three in ",[194,284,285],{},"requirements.txt"," and treat a scheduler upgrade as a real migration with a test run, not a passive ",[194,288,289],{},"pip install --upgrade",".",[163,292],{},[166,294,296],{"id":295},"step-1-write-one-idempotent-pipeline-function","Step 1 — Write one idempotent pipeline function",[14,298,299,300,303],{},"Everything downstream calls ",[27,301,302],{},"this",". It reads config from the environment, does extract → transform → load, and is safe to run twice. Idempotency is what makes scheduling forgiving: a missed run, a manual re-run, or an overlapping trigger should never corrupt your data. It is also what lets you downgrade the guarantee your scheduler has to provide — and cheaper schedulers provide weaker guarantees.",[187,305,309],{"className":306,"code":307,"language":308,"meta":192,"style":192},"language-python shiki shiki-themes github-light github-dark","import os\nimport time\nimport httpx\n\ndef run_pipeline() -> dict:\n    \"\"\"Idempotent extract-transform-load. Safe to run more than once.\"\"\"\n    name = os.getenv(\"PIPELINE_NAME\", \"pipeline\")\n    api_url = os.environ[\"SOURCE_API_URL\"]\n    api_key = os.environ[\"SOURCE_API_KEY\"]\n\n    started = time.perf_counter()\n\n    # Extract\n    with httpx.Client(timeout=30.0) as client:\n        resp = client.get(api_url, headers={\"Authorization\": f\"Bearer {api_key}\"})\n        resp.raise_for_status()\n        rows = resp.json().get(\"orders\", [])\n\n    # Transform\n    cleaned = [\n        {\"id\": r[\"id\"], \"total\": float(r[\"total_price\"])}\n        for r in rows\n        if r.get(\"financial_status\") == \"paid\"\n    ]\n\n    # Load — keyed by id so re-running upserts instead of duplicating\n    upserted = upsert_by_id(cleaned)\n\n    return {\n        \"pipeline\": name,\n        \"fetched\": len(rows),\n        \"loaded\": upserted,\n        \"duration_s\": round(time.perf_counter() - started, 3),\n    }\n","python",[194,310,311,321,328,335,341,359,364,387,403,418,423,434,439,446,473,518,524,541,546,552,563,597,612,632,638,643,649,660,665,674,683,697,706,732],{"__ignoreMap":192},[197,312,313,317],{"class":77,"line":199},[197,314,316],{"class":315},"szBVR","import",[197,318,320],{"class":319},"sVt8B"," os\n",[197,322,323,325],{"class":77,"line":240},[197,324,316],{"class":315},[197,326,327],{"class":319}," time\n",[197,329,330,332],{"class":77,"line":246},[197,331,316],{"class":315},[197,333,334],{"class":319}," httpx\n",[197,336,337],{"class":77,"line":252},[197,338,340],{"emptyLinePlaceholder":339},true,"\n",[197,342,343,346,349,352,356],{"class":77,"line":258},[197,344,345],{"class":315},"def",[197,347,348],{"class":202}," run_pipeline",[197,350,351],{"class":319},"() -> ",[197,353,355],{"class":354},"sj4cs","dict",[197,357,358],{"class":319},":\n",[197,360,361],{"class":77,"line":264},[197,362,363],{"class":206},"    \"\"\"Idempotent extract-transform-load. Safe to run more than once.\"\"\"\n",[197,365,366,369,372,375,378,381,384],{"class":77,"line":270},[197,367,368],{"class":319},"    name ",[197,370,371],{"class":315},"=",[197,373,374],{"class":319}," os.getenv(",[197,376,377],{"class":206},"\"PIPELINE_NAME\"",[197,379,380],{"class":319},", ",[197,382,383],{"class":206},"\"pipeline\"",[197,385,386],{"class":319},")\n",[197,388,389,392,394,397,400],{"class":77,"line":276},[197,390,391],{"class":319},"    api_url ",[197,393,371],{"class":315},[197,395,396],{"class":319}," os.environ[",[197,398,399],{"class":206},"\"SOURCE_API_URL\"",[197,401,402],{"class":319},"]\n",[197,404,406,409,411,413,416],{"class":77,"line":405},9,[197,407,408],{"class":319},"    api_key ",[197,410,371],{"class":315},[197,412,396],{"class":319},[197,414,415],{"class":206},"\"SOURCE_API_KEY\"",[197,417,402],{"class":319},[197,419,421],{"class":77,"line":420},10,[197,422,340],{"emptyLinePlaceholder":339},[197,424,426,429,431],{"class":77,"line":425},11,[197,427,428],{"class":319},"    started ",[197,430,371],{"class":315},[197,432,433],{"class":319}," time.perf_counter()\n",[197,435,437],{"class":77,"line":436},12,[197,438,340],{"emptyLinePlaceholder":339},[197,440,442],{"class":77,"line":441},13,[197,443,445],{"class":444},"sJ8bj","    # Extract\n",[197,447,449,452,455,459,461,464,467,470],{"class":77,"line":448},14,[197,450,451],{"class":315},"    with",[197,453,454],{"class":319}," httpx.Client(",[197,456,458],{"class":457},"s4XuR","timeout",[197,460,371],{"class":315},[197,462,463],{"class":354},"30.0",[197,465,466],{"class":319},") ",[197,468,469],{"class":315},"as",[197,471,472],{"class":319}," client:\n",[197,474,476,479,481,484,487,489,492,495,498,501,504,506,509,512,515],{"class":77,"line":475},15,[197,477,478],{"class":319},"        resp ",[197,480,371],{"class":315},[197,482,483],{"class":319}," client.get(api_url, ",[197,485,486],{"class":457},"headers",[197,488,371],{"class":315},[197,490,491],{"class":319},"{",[197,493,494],{"class":206},"\"Authorization\"",[197,496,497],{"class":319},": ",[197,499,500],{"class":315},"f",[197,502,503],{"class":206},"\"Bearer ",[197,505,491],{"class":354},[197,507,508],{"class":319},"api_key",[197,510,511],{"class":354},"}",[197,513,514],{"class":206},"\"",[197,516,517],{"class":319},"})\n",[197,519,521],{"class":77,"line":520},16,[197,522,523],{"class":319},"        resp.raise_for_status()\n",[197,525,527,530,532,535,538],{"class":77,"line":526},17,[197,528,529],{"class":319},"        rows ",[197,531,371],{"class":315},[197,533,534],{"class":319}," resp.json().get(",[197,536,537],{"class":206},"\"orders\"",[197,539,540],{"class":319},", [])\n",[197,542,544],{"class":77,"line":543},18,[197,545,340],{"emptyLinePlaceholder":339},[197,547,549],{"class":77,"line":548},19,[197,550,551],{"class":444},"    # Transform\n",[197,553,555,558,560],{"class":77,"line":554},20,[197,556,557],{"class":319},"    cleaned ",[197,559,371],{"class":315},[197,561,562],{"class":319}," [\n",[197,564,566,569,572,575,577,580,583,585,588,591,594],{"class":77,"line":565},21,[197,567,568],{"class":319},"        {",[197,570,571],{"class":206},"\"id\"",[197,573,574],{"class":319},": r[",[197,576,571],{"class":206},[197,578,579],{"class":319},"], ",[197,581,582],{"class":206},"\"total\"",[197,584,497],{"class":319},[197,586,587],{"class":354},"float",[197,589,590],{"class":319},"(r[",[197,592,593],{"class":206},"\"total_price\"",[197,595,596],{"class":319},"])}\n",[197,598,600,603,606,609],{"class":77,"line":599},22,[197,601,602],{"class":315},"        for",[197,604,605],{"class":319}," r ",[197,607,608],{"class":315},"in",[197,610,611],{"class":319}," rows\n",[197,613,615,618,621,624,626,629],{"class":77,"line":614},23,[197,616,617],{"class":315},"        if",[197,619,620],{"class":319}," r.get(",[197,622,623],{"class":206},"\"financial_status\"",[197,625,466],{"class":319},[197,627,628],{"class":315},"==",[197,630,631],{"class":206}," \"paid\"\n",[197,633,635],{"class":77,"line":634},24,[197,636,637],{"class":319},"    ]\n",[197,639,641],{"class":77,"line":640},25,[197,642,340],{"emptyLinePlaceholder":339},[197,644,646],{"class":77,"line":645},26,[197,647,648],{"class":444},"    # Load — keyed by id so re-running upserts instead of duplicating\n",[197,650,652,655,657],{"class":77,"line":651},27,[197,653,654],{"class":319},"    upserted ",[197,656,371],{"class":315},[197,658,659],{"class":319}," upsert_by_id(cleaned)\n",[197,661,663],{"class":77,"line":662},28,[197,664,340],{"emptyLinePlaceholder":339},[197,666,668,671],{"class":77,"line":667},29,[197,669,670],{"class":315},"    return",[197,672,673],{"class":319}," {\n",[197,675,677,680],{"class":77,"line":676},30,[197,678,679],{"class":206},"        \"pipeline\"",[197,681,682],{"class":319},": name,\n",[197,684,686,689,691,694],{"class":77,"line":685},31,[197,687,688],{"class":206},"        \"fetched\"",[197,690,497],{"class":319},[197,692,693],{"class":354},"len",[197,695,696],{"class":319},"(rows),\n",[197,698,700,703],{"class":77,"line":699},32,[197,701,702],{"class":206},"        \"loaded\"",[197,704,705],{"class":319},": upserted,\n",[197,707,709,712,714,717,720,723,726,729],{"class":77,"line":708},33,[197,710,711],{"class":206},"        \"duration_s\"",[197,713,497],{"class":319},[197,715,716],{"class":354},"round",[197,718,719],{"class":319},"(time.perf_counter() ",[197,721,722],{"class":315},"-",[197,724,725],{"class":319}," started, ",[197,727,728],{"class":354},"3",[197,730,731],{"class":319},"),\n",[197,733,735],{"class":77,"line":734},34,[197,736,737],{"class":319},"    }\n",[14,739,740,741,744,745,748,749,753],{},"The ",[194,742,743],{},"upsert_by_id"," helper is the single most important line in the file. In Postgres it is an ",[194,746,747],{},"INSERT ... ON CONFLICT (id) DO UPDATE","; against a Google Sheet it is a set-difference against existing keys before you append. Whatever the backing store, it turns \"run exactly once\" — a distributed-systems problem nobody fully solves — into \"run at least once,\" which every scheduler on earth can guarantee for free. If your load target is a real database, wire the upsert through an async session as described in ",[18,750,752],{"href":751},"\u002Fscaling-and-operating-production-python-apis\u002Fasync-database-access-with-sqlalchemy\u002F","async database access with SQLAlchemy"," so the pipeline shares the same connection pool as the rest of your app instead of opening a fresh connection every night.",[14,755,756,757,760,761,764,765,768],{},"One thing to resist: do not make the transform depend on ",[27,758,759],{},"when"," it runs. A pipeline that reads \"yesterday\" from the wall clock behaves differently on a re-run than on the original run, and that non-determinism is the enemy of idempotency. Pass the window in explicitly — a ",[194,762,763],{},"since"," timestamp derived from the last successful watermark you stored, not from ",[194,766,767],{},"datetime.now()"," — so a re-run of a missed night processes the night it missed, not the night you happen to re-run it.",[163,770],{},[166,772,774],{"id":773},"step-2-choose-the-scheduler-that-fits-your-stage","Step 2 — Choose the scheduler that fits your stage",[14,776,777],{},"Before writing any scheduling code, decide which clock you actually need. The decision is not about which library is \"best\" — it is about whether you already pay for an always-on process, how precise your timing must be, and whether the job has to coordinate across more than one machine. Get this right and you avoid paying for infrastructure you do not use.",[32,779,41,784,41,787,41,790,41,797,41,800,41,803,41,806,41,812,41,816,41,820,41,824,41,827,41,830,41,832,41,835,41,840,41,843,41,845,41,848,41,851,41,854,41,857,41,860,41,866,41,870,41,873,41,876,41,879,41,882,41,885,41,888,41,891,41,894,41,897,41,900,41,904],{"viewBox":780,"role":35,"ariaLabelledBy":781,"xmlns":39,"style":40},"0 0 720 340",[782,783],"cron-tree-t","cron-tree-d",[43,785,786],{"id":782},"Decision tree for choosing a Python scheduler",[47,788,789],{"id":783},"Branches on whether you run an always-on process, need minute precision, and run across multiple machines, ending at managed cron, OS cron, APScheduler, or Celery beat.",[51,791,53,792,41],{},[55,793,63,795,53],{"id":794,"viewBox":58,"refX":59,"refY":60,"markerWidth":61,"markerHeight":61,"orient":62},"cron-tree-arrow",[65,796],{"d":67,"fill":68},[70,798],{"x":72,"y":72,"width":73,"height":799,"fill":75},"340",[70,801],{"x":74,"y":802,"width":128,"height":80,"rx":117,"fill":75,"stroke":118,"style":83},"18",[100,804,805],{"x":102,"y":80,"fill":123,"style":104},"Already run an always-on app?",[77,807],{"x1":808,"y1":809,"x2":115,"y2":810,"stroke":68,"style":811},"300","64","98","stroke-width:2;marker-end:url(#cron-tree-arrow);",[100,813,815],{"x":151,"y":814,"fill":68,"style":160},"86","no",[77,817],{"x1":818,"y1":809,"x2":819,"y2":810,"stroke":68,"style":811},"420","540",[100,821,823],{"x":822,"y":814,"fill":68,"style":160},"508","yes",[70,825],{"x":116,"y":826,"width":154,"height":80,"rx":117,"fill":75,"stroke":82,"style":83},"102",[100,828,829],{"x":115,"y":121,"fill":123,"style":104},"Need minute precision?",[70,831],{"x":818,"y":826,"width":154,"height":80,"rx":117,"fill":75,"stroke":82,"style":83},[100,833,834],{"x":819,"y":121,"fill":123,"style":104},"Run across many machines?",[77,836],{"x1":87,"y1":837,"x2":838,"y2":839,"stroke":68,"style":811},"148","90","196",[100,841,815],{"x":814,"y":842,"fill":68,"style":160},"176",[77,844],{"x1":154,"y1":837,"x2":130,"y2":839,"stroke":68,"style":811},[100,846,823],{"x":847,"y":842,"fill":68,"style":160},"272",[77,849],{"x1":850,"y1":837,"x2":139,"y2":839,"stroke":68,"style":811},"480",[100,852,815],{"x":853,"y":842,"fill":68,"style":160},"446",[77,855],{"x1":98,"y1":837,"x2":856,"y2":839,"stroke":68,"style":811},"630",[100,858,823],{"x":859,"y":842,"fill":68,"style":160},"632",[70,861],{"x":862,"y":863,"width":864,"height":865,"rx":117,"fill":75,"stroke":133,"style":83},"24","200","132","52",[100,867,869],{"x":838,"y":868,"fill":123,"style":104},"223","Managed",[100,871,872],{"x":838,"y":154,"fill":123,"style":104},"cron",[70,874],{"x":875,"y":863,"width":864,"height":865,"rx":117,"fill":75,"stroke":133,"style":83},"204",[100,877,878],{"x":130,"y":868,"fill":123,"style":104},"OS cron",[100,880,881],{"x":130,"y":154,"fill":123,"style":104},"+ flock",[70,883],{"x":884,"y":863,"width":864,"height":865,"rx":117,"fill":75,"stroke":143,"style":83},"384",[100,886,887],{"x":139,"y":868,"fill":123,"style":104},"APScheduler",[100,889,890],{"x":139,"y":154,"fill":123,"style":104},"in-process",[70,892],{"x":893,"y":863,"width":864,"height":865,"rx":117,"fill":75,"stroke":143,"style":83},"564",[100,895,896],{"x":856,"y":868,"fill":123,"style":104},"Celery",[100,898,899],{"x":856,"y":154,"fill":123,"style":104},"beat",[100,901,903],{"x":102,"y":902,"fill":68,"style":160},"292","Pay-per-run on the left; always-on infrastructure on the right.",[100,905,907],{"x":102,"y":906,"fill":68,"style":160},"314","Move rightward only when the business forces you to.",[14,909,910,911,915],{},"Read the tree from the top. Most side hustles land on the left two boxes and should stay there for a long time. You move right only when a concrete need forces it: minute-level precision (a price scraper that must beat a competitor's cache), or coordination across machines (you already run a fleet of ",[18,912,914],{"href":913},"\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002F","Celery background workers"," and adding a second scheduler would be silly). The rest of this guide implements each leaf so you can see the real code behind the choice.",[163,917],{},[166,919,921],{"id":920},"step-3-run-it-in-process-with-apscheduler","Step 3 — Run it in-process with APScheduler",[14,923,924,925,928],{},"For a single always-on container, APScheduler is the lightest option: no broker, no extra process. The ",[194,926,927],{},"AsyncIOScheduler"," fires your job on a cron expression inside your app's event loop, which makes it a natural fit when you have already deployed a FastAPI service and just want it to also do overnight work.",[187,930,932],{"className":306,"code":931,"language":308,"meta":192,"style":192},"import asyncio\nimport os\nfrom apscheduler.schedulers.asyncio import AsyncIOScheduler\nfrom apscheduler.triggers.cron import CronTrigger\n\nasync def scheduled_job() -> None:\n    # Run blocking pipeline off the event loop\n    result = await asyncio.to_thread(run_pipeline)\n    print(result)\n\nasync def main() -> None:\n    scheduler = AsyncIOScheduler(timezone=os.getenv(\"PIPELINE_TZ\", \"UTC\"))\n    trigger = CronTrigger.from_crontab(\n        os.getenv(\"CRON_SCHEDULE\", \"0 2 * * *\"),\n        timezone=os.getenv(\"PIPELINE_TZ\", \"UTC\"),\n    )\n    scheduler.add_job(\n        scheduled_job,\n        trigger=trigger,\n        id=\"nightly-pipeline\",\n        max_instances=1,        # never overlap\n        coalesce=True,          # collapse missed runs into one\n        misfire_grace_time=3600,\n    )\n    scheduler.start()\n    await asyncio.Event().wait()  # keep the loop alive\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n",[194,933,934,941,947,960,972,976,994,999,1012,1020,1024,1039,1068,1078,1093,1110,1115,1120,1125,1135,1148,1164,1180,1192,1196,1201,1212,1216,1232],{"__ignoreMap":192},[197,935,936,938],{"class":77,"line":199},[197,937,316],{"class":315},[197,939,940],{"class":319}," asyncio\n",[197,942,943,945],{"class":77,"line":240},[197,944,316],{"class":315},[197,946,320],{"class":319},[197,948,949,952,955,957],{"class":77,"line":246},[197,950,951],{"class":315},"from",[197,953,954],{"class":319}," apscheduler.schedulers.asyncio ",[197,956,316],{"class":315},[197,958,959],{"class":319}," AsyncIOScheduler\n",[197,961,962,964,967,969],{"class":77,"line":252},[197,963,951],{"class":315},[197,965,966],{"class":319}," apscheduler.triggers.cron ",[197,968,316],{"class":315},[197,970,971],{"class":319}," CronTrigger\n",[197,973,974],{"class":77,"line":258},[197,975,340],{"emptyLinePlaceholder":339},[197,977,978,981,984,987,989,992],{"class":77,"line":264},[197,979,980],{"class":315},"async",[197,982,983],{"class":315}," def",[197,985,986],{"class":202}," scheduled_job",[197,988,351],{"class":319},[197,990,991],{"class":354},"None",[197,993,358],{"class":319},[197,995,996],{"class":77,"line":270},[197,997,998],{"class":444},"    # Run blocking pipeline off the event loop\n",[197,1000,1001,1004,1006,1009],{"class":77,"line":276},[197,1002,1003],{"class":319},"    result ",[197,1005,371],{"class":315},[197,1007,1008],{"class":315}," await",[197,1010,1011],{"class":319}," asyncio.to_thread(run_pipeline)\n",[197,1013,1014,1017],{"class":77,"line":405},[197,1015,1016],{"class":354},"    print",[197,1018,1019],{"class":319},"(result)\n",[197,1021,1022],{"class":77,"line":420},[197,1023,340],{"emptyLinePlaceholder":339},[197,1025,1026,1028,1030,1033,1035,1037],{"class":77,"line":425},[197,1027,980],{"class":315},[197,1029,983],{"class":315},[197,1031,1032],{"class":202}," main",[197,1034,351],{"class":319},[197,1036,991],{"class":354},[197,1038,358],{"class":319},[197,1040,1041,1044,1046,1049,1052,1054,1057,1060,1062,1065],{"class":77,"line":436},[197,1042,1043],{"class":319},"    scheduler ",[197,1045,371],{"class":315},[197,1047,1048],{"class":319}," AsyncIOScheduler(",[197,1050,1051],{"class":457},"timezone",[197,1053,371],{"class":315},[197,1055,1056],{"class":319},"os.getenv(",[197,1058,1059],{"class":206},"\"PIPELINE_TZ\"",[197,1061,380],{"class":319},[197,1063,1064],{"class":206},"\"UTC\"",[197,1066,1067],{"class":319},"))\n",[197,1069,1070,1073,1075],{"class":77,"line":441},[197,1071,1072],{"class":319},"    trigger ",[197,1074,371],{"class":315},[197,1076,1077],{"class":319}," CronTrigger.from_crontab(\n",[197,1079,1080,1083,1086,1088,1091],{"class":77,"line":448},[197,1081,1082],{"class":319},"        os.getenv(",[197,1084,1085],{"class":206},"\"CRON_SCHEDULE\"",[197,1087,380],{"class":319},[197,1089,1090],{"class":206},"\"0 2 * * *\"",[197,1092,731],{"class":319},[197,1094,1095,1098,1100,1102,1104,1106,1108],{"class":77,"line":475},[197,1096,1097],{"class":457},"        timezone",[197,1099,371],{"class":315},[197,1101,1056],{"class":319},[197,1103,1059],{"class":206},[197,1105,380],{"class":319},[197,1107,1064],{"class":206},[197,1109,731],{"class":319},[197,1111,1112],{"class":77,"line":520},[197,1113,1114],{"class":319},"    )\n",[197,1116,1117],{"class":77,"line":526},[197,1118,1119],{"class":319},"    scheduler.add_job(\n",[197,1121,1122],{"class":77,"line":543},[197,1123,1124],{"class":319},"        scheduled_job,\n",[197,1126,1127,1130,1132],{"class":77,"line":548},[197,1128,1129],{"class":457},"        trigger",[197,1131,371],{"class":315},[197,1133,1134],{"class":319},"trigger,\n",[197,1136,1137,1140,1142,1145],{"class":77,"line":554},[197,1138,1139],{"class":457},"        id",[197,1141,371],{"class":315},[197,1143,1144],{"class":206},"\"nightly-pipeline\"",[197,1146,1147],{"class":319},",\n",[197,1149,1150,1153,1155,1158,1161],{"class":77,"line":565},[197,1151,1152],{"class":457},"        max_instances",[197,1154,371],{"class":315},[197,1156,1157],{"class":354},"1",[197,1159,1160],{"class":319},",        ",[197,1162,1163],{"class":444},"# never overlap\n",[197,1165,1166,1169,1171,1174,1177],{"class":77,"line":599},[197,1167,1168],{"class":457},"        coalesce",[197,1170,371],{"class":315},[197,1172,1173],{"class":354},"True",[197,1175,1176],{"class":319},",          ",[197,1178,1179],{"class":444},"# collapse missed runs into one\n",[197,1181,1182,1185,1187,1190],{"class":77,"line":614},[197,1183,1184],{"class":457},"        misfire_grace_time",[197,1186,371],{"class":315},[197,1188,1189],{"class":354},"3600",[197,1191,1147],{"class":319},[197,1193,1194],{"class":77,"line":634},[197,1195,1114],{"class":319},[197,1197,1198],{"class":77,"line":640},[197,1199,1200],{"class":319},"    scheduler.start()\n",[197,1202,1203,1206,1209],{"class":77,"line":645},[197,1204,1205],{"class":315},"    await",[197,1207,1208],{"class":319}," asyncio.Event().wait()  ",[197,1210,1211],{"class":444},"# keep the loop alive\n",[197,1213,1214],{"class":77,"line":651},[197,1215,340],{"emptyLinePlaceholder":339},[197,1217,1218,1221,1224,1227,1230],{"class":77,"line":662},[197,1219,1220],{"class":315},"if",[197,1222,1223],{"class":354}," __name__",[197,1225,1226],{"class":315}," ==",[197,1228,1229],{"class":206}," \"__main__\"",[197,1231,358],{"class":319},[197,1233,1234],{"class":77,"line":667},[197,1235,1236],{"class":319},"    asyncio.run(main())\n",[14,1238,1239,1240,1243,1244,1247,1248,1251,1252,1255,1256,1259,1260,1263],{},"Three keyword arguments carry the entire reliability story. ",[194,1241,1242],{},"max_instances=1"," guarantees the job never overlaps with itself — critical when a slow night runs past the next trigger. ",[194,1245,1246],{},"coalesce=True"," collapses a backlog of missed triggers (the container was asleep) into a single catch-up run instead of firing six times in a row. ",[194,1249,1250],{},"misfire_grace_time=3600"," says a trigger is still worth honouring up to an hour late, after which APScheduler quietly drops it rather than running a nightly job at noon. The ",[194,1253,1254],{},"asyncio.to_thread"," call matters just as much: ",[194,1257,1258],{},"run_pipeline"," uses a synchronous ",[194,1261,1262],{},"httpx.Client",", and calling it directly on the event loop would freeze every other request your app is serving for the duration of the sync. Offloading it to a thread keeps the loop responsive.",[14,1265,1266,1267,1270,1271,1274,1275,1279],{},"APScheduler's weakness is that its schedule lives in memory. If the process dies, the schedule dies with it, and a job that was due while the process was down is gone unless ",[194,1268,1269],{},"coalesce"," and ",[194,1272,1273],{},"misfire_grace_time"," cover the gap. If you are weighing this against a worker-based scheduler, read ",[18,1276,1278],{"href":1277},"\u002Fautomating-side-hustle-operations-with-apis\u002Fscheduling-data-pipelines-with-cron\u002Fapscheduler-vs-celery-beat\u002F","APScheduler vs Celery beat"," before committing — it walks the durability trade-off in detail.",[163,1281],{},[166,1283,1285],{"id":1284},"step-4-run-the-same-task-from-os-cron-with-a-lock","Step 4 — Run the same task from OS cron with a lock",[14,1287,1288,1289,1292],{},"If you do not want an always-on process, OS cron is free and battle-tested. It invokes a script on a schedule; you supply the entrypoint. The catch: cron will happily start a second copy while the first is still running, and unlike APScheduler it has no ",[194,1290,1291],{},"max_instances",". A file lock supplies that guarantee.",[187,1294,1296],{"className":306,"code":1295,"language":308,"meta":192,"style":192},"# pipeline_entrypoint.py\nimport os\nimport sys\nimport fcntl\nfrom contextlib import contextmanager\n\n@contextmanager\ndef single_run_lock(path: str):\n    fd = open(path, \"w\")\n    try:\n        fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)\n    except BlockingIOError:\n        print(\"Previous run still active; skipping.\", file=sys.stderr)\n        sys.exit(0)\n    try:\n        yield\n    finally:\n        fcntl.flock(fd, fcntl.LOCK_UN)\n        fd.close()\n\nif __name__ == \"__main__\":\n    lock_path = os.getenv(\"LOCK_PATH\", \"\u002Ftmp\u002Fpipeline.lock\")\n    with single_run_lock(lock_path):\n        print(run_pipeline())\n",[194,1297,1298,1303,1309,1316,1323,1335,1339,1344,1360,1378,1385,1404,1414,1435,1444,1450,1455,1462,1471,1476,1480,1492,1511,1518],{"__ignoreMap":192},[197,1299,1300],{"class":77,"line":199},[197,1301,1302],{"class":444},"# pipeline_entrypoint.py\n",[197,1304,1305,1307],{"class":77,"line":240},[197,1306,316],{"class":315},[197,1308,320],{"class":319},[197,1310,1311,1313],{"class":77,"line":246},[197,1312,316],{"class":315},[197,1314,1315],{"class":319}," sys\n",[197,1317,1318,1320],{"class":77,"line":252},[197,1319,316],{"class":315},[197,1321,1322],{"class":319}," fcntl\n",[197,1324,1325,1327,1330,1332],{"class":77,"line":258},[197,1326,951],{"class":315},[197,1328,1329],{"class":319}," contextlib ",[197,1331,316],{"class":315},[197,1333,1334],{"class":319}," contextmanager\n",[197,1336,1337],{"class":77,"line":264},[197,1338,340],{"emptyLinePlaceholder":339},[197,1340,1341],{"class":77,"line":270},[197,1342,1343],{"class":202},"@contextmanager\n",[197,1345,1346,1348,1351,1354,1357],{"class":77,"line":276},[197,1347,345],{"class":315},[197,1349,1350],{"class":202}," single_run_lock",[197,1352,1353],{"class":319},"(path: ",[197,1355,1356],{"class":354},"str",[197,1358,1359],{"class":319},"):\n",[197,1361,1362,1365,1367,1370,1373,1376],{"class":77,"line":405},[197,1363,1364],{"class":319},"    fd ",[197,1366,371],{"class":315},[197,1368,1369],{"class":354}," open",[197,1371,1372],{"class":319},"(path, ",[197,1374,1375],{"class":206},"\"w\"",[197,1377,386],{"class":319},[197,1379,1380,1383],{"class":77,"line":420},[197,1381,1382],{"class":315},"    try",[197,1384,358],{"class":319},[197,1386,1387,1390,1393,1396,1399,1402],{"class":77,"line":425},[197,1388,1389],{"class":319},"        fcntl.flock(fd, fcntl.",[197,1391,1392],{"class":354},"LOCK_EX",[197,1394,1395],{"class":315}," |",[197,1397,1398],{"class":319}," fcntl.",[197,1400,1401],{"class":354},"LOCK_NB",[197,1403,386],{"class":319},[197,1405,1406,1409,1412],{"class":77,"line":436},[197,1407,1408],{"class":315},"    except",[197,1410,1411],{"class":354}," BlockingIOError",[197,1413,358],{"class":319},[197,1415,1416,1419,1422,1425,1427,1430,1432],{"class":77,"line":441},[197,1417,1418],{"class":354},"        print",[197,1420,1421],{"class":319},"(",[197,1423,1424],{"class":206},"\"Previous run still active; skipping.\"",[197,1426,380],{"class":319},[197,1428,1429],{"class":457},"file",[197,1431,371],{"class":315},[197,1433,1434],{"class":319},"sys.stderr)\n",[197,1436,1437,1440,1442],{"class":77,"line":448},[197,1438,1439],{"class":319},"        sys.exit(",[197,1441,72],{"class":354},[197,1443,386],{"class":319},[197,1445,1446,1448],{"class":77,"line":475},[197,1447,1382],{"class":315},[197,1449,358],{"class":319},[197,1451,1452],{"class":77,"line":520},[197,1453,1454],{"class":315},"        yield\n",[197,1456,1457,1460],{"class":77,"line":526},[197,1458,1459],{"class":315},"    finally",[197,1461,358],{"class":319},[197,1463,1464,1466,1469],{"class":77,"line":543},[197,1465,1389],{"class":319},[197,1467,1468],{"class":354},"LOCK_UN",[197,1470,386],{"class":319},[197,1472,1473],{"class":77,"line":548},[197,1474,1475],{"class":319},"        fd.close()\n",[197,1477,1478],{"class":77,"line":554},[197,1479,340],{"emptyLinePlaceholder":339},[197,1481,1482,1484,1486,1488,1490],{"class":77,"line":565},[197,1483,1220],{"class":315},[197,1485,1223],{"class":354},[197,1487,1226],{"class":315},[197,1489,1229],{"class":206},[197,1491,358],{"class":319},[197,1493,1494,1497,1499,1501,1504,1506,1509],{"class":77,"line":599},[197,1495,1496],{"class":319},"    lock_path ",[197,1498,371],{"class":315},[197,1500,374],{"class":319},[197,1502,1503],{"class":206},"\"LOCK_PATH\"",[197,1505,380],{"class":319},[197,1507,1508],{"class":206},"\"\u002Ftmp\u002Fpipeline.lock\"",[197,1510,386],{"class":319},[197,1512,1513,1515],{"class":77,"line":614},[197,1514,451],{"class":315},[197,1516,1517],{"class":319}," single_run_lock(lock_path):\n",[197,1519,1520,1522],{"class":77,"line":634},[197,1521,1418],{"class":354},[197,1523,1524],{"class":319},"(run_pipeline())\n",[14,1526,1527],{},"The crontab entry loads the environment and calls the entrypoint:",[187,1529,1532],{"className":1530,"code":1531,"language":872,"meta":192,"style":192},"language-cron shiki shiki-themes github-light github-dark","0 2 * * * cd \u002Fsrv\u002Fapp && \u002Fsrv\u002Fapp\u002F.venv\u002Fbin\u002Fpython pipeline_entrypoint.py >> \u002Fvar\u002Flog\u002Fpipeline.log 2>&1\n",[194,1533,1534],{"__ignoreMap":192},[197,1535,1536],{"class":77,"line":199},[197,1537,1531],{},[14,1539,1540,1543,1544,1547,1548,1551,1552,1554],{},[194,1541,1542],{},"LOCK_EX | LOCK_NB"," means \"grab the lock or fail immediately\" — exactly what overlap protection requires. The ",[194,1545,1546],{},"flock"," is advisory and tied to the file descriptor, so it releases automatically if the process crashes; there is no stale lock to clean up by hand, which is the failure mode that plagues homegrown \"write a PID file\" schemes. Two footguns to remember. First, cron does not load your shell profile, so ",[194,1549,1550],{},"SOURCE_API_KEY"," will be empty unless you source an env file inside the script or set variables in the crontab itself. Second, ",[194,1553,1546],{}," on a local file only guards a single machine; if you run cron on two servers pointing at the same database, each has its own lock and both will run. The diagram below shows exactly what the lock buys you when a slow run overruns its next trigger.",[32,1556,41,1561,41,1564,41,1567,41,1570,41,1574,41,1577,41,1580,41,1585,41,1588,41,1592,41,1595,41,1598,41,1600,41,1604,41,1606,41,1609,41,1613],{"viewBox":1557,"role":35,"ariaLabelledBy":1558,"xmlns":39,"style":40},"0 0 720 260",[1559,1560],"cron-lock-t","cron-lock-d",[43,1562,1563],{"id":1559},"How a file lock skips an overlapping cron trigger",[47,1565,1566],{"id":1560},"Two nightly triggers fire while a long run is in progress; the lock lets the first run proceed and causes the second trigger to exit immediately instead of overlapping.",[70,1568],{"x":72,"y":72,"width":73,"height":1569,"fill":75},"260",[100,1571,1573],{"x":862,"y":113,"fill":68,"style":1572},"text-anchor:start;font-size:12;font-family:var(--font-sans);","02:00 trigger",[100,1575,1576],{"x":862,"y":114,"fill":68,"style":1572},"Run A (long)",[100,1578,1579],{"x":862,"y":115,"fill":68,"style":1572},"02:30 trigger",[77,1581],{"x1":1582,"y1":862,"x2":1582,"y2":1583,"stroke":82,"style":1584},"150","236","stroke-width:1;",[77,1586],{"x1":1587,"y1":862,"x2":1587,"y2":1583,"stroke":82,"style":1584},"470",[100,1589,1591],{"x":1582,"y":802,"fill":68,"style":1590},"text-anchor:middle;font-size:11;font-family:var(--font-sans);","02:00",[100,1593,1594],{"x":1587,"y":802,"fill":68,"style":1590},"02:30",[85,1596],{"cx":1582,"cy":1597,"r":88,"fill":89},"36",[70,1599],{"x":1582,"y":838,"width":95,"height":79,"rx":156,"fill":75,"stroke":118,"style":83},[100,1601,1603],{"x":1602,"y":114,"fill":123,"style":160},"370","holds lock, keeps running",[85,1605],{"cx":1587,"cy":150,"r":88,"fill":89},[70,1607],{"x":1587,"y":1608,"width":1582,"height":79,"rx":156,"fill":75,"stroke":143,"style":83},"156",[100,1610,1612],{"x":1611,"y":842,"fill":123,"style":160},"545","lock busy, exits 0",[100,1614,1616],{"x":102,"y":1615,"fill":68,"style":160},"228","No overlap: the second trigger skips instead of corrupting shared data.",[163,1618],{},[166,1620,1622],{"id":1621},"step-5-run-it-distributed-with-celery-beat","Step 5 — Run it distributed with Celery beat",[14,1624,1625,1626,1629,1630,1632],{},"Once you already run ",[18,1627,1628],{"href":913},"background workers with Celery"," for webhook processing or long jobs, do not bolt on a second scheduler. Celery beat is a tick process that enqueues your task on a schedule; an existing worker executes it. The overlap guard now lives in the broker, so multiple workers across machines still run the job once — the exact guarantee ",[194,1631,1546],{}," cannot give you across servers.",[187,1634,1636],{"className":306,"code":1635,"language":308,"meta":192,"style":192},"import os\nfrom celery import Celery\nfrom celery.schedules import crontab\n\napp = Celery(\"hustle\", broker=os.environ[\"CELERY_BROKER_URL\"])\n\n@app.task(bind=True, max_retries=3)\ndef pipeline_task(self):\n    try:\n        return run_pipeline()\n    except Exception as exc:\n        raise self.retry(exc=exc, countdown=60)\n\napp.conf.beat_schedule = {\n    \"nightly-pipeline\": {\n        \"task\": \"pipeline_task\",\n        \"schedule\": crontab(hour=2, minute=0),\n    }\n}\napp.conf.timezone = os.getenv(\"PIPELINE_TZ\", \"UTC\")\n",[194,1637,1638,1644,1656,1668,1672,1701,1705,1730,1740,1746,1754,1767,1795,1799,1808,1816,1828,1855,1859,1864],{"__ignoreMap":192},[197,1639,1640,1642],{"class":77,"line":199},[197,1641,316],{"class":315},[197,1643,320],{"class":319},[197,1645,1646,1648,1651,1653],{"class":77,"line":240},[197,1647,951],{"class":315},[197,1649,1650],{"class":319}," celery ",[197,1652,316],{"class":315},[197,1654,1655],{"class":319}," Celery\n",[197,1657,1658,1660,1663,1665],{"class":77,"line":246},[197,1659,951],{"class":315},[197,1661,1662],{"class":319}," celery.schedules ",[197,1664,316],{"class":315},[197,1666,1667],{"class":319}," crontab\n",[197,1669,1670],{"class":77,"line":252},[197,1671,340],{"emptyLinePlaceholder":339},[197,1673,1674,1677,1679,1682,1685,1687,1690,1692,1695,1698],{"class":77,"line":258},[197,1675,1676],{"class":319},"app ",[197,1678,371],{"class":315},[197,1680,1681],{"class":319}," Celery(",[197,1683,1684],{"class":206},"\"hustle\"",[197,1686,380],{"class":319},[197,1688,1689],{"class":457},"broker",[197,1691,371],{"class":315},[197,1693,1694],{"class":319},"os.environ[",[197,1696,1697],{"class":206},"\"CELERY_BROKER_URL\"",[197,1699,1700],{"class":319},"])\n",[197,1702,1703],{"class":77,"line":264},[197,1704,340],{"emptyLinePlaceholder":339},[197,1706,1707,1710,1712,1715,1717,1719,1721,1724,1726,1728],{"class":77,"line":270},[197,1708,1709],{"class":202},"@app.task",[197,1711,1421],{"class":319},[197,1713,1714],{"class":457},"bind",[197,1716,371],{"class":315},[197,1718,1173],{"class":354},[197,1720,380],{"class":319},[197,1722,1723],{"class":457},"max_retries",[197,1725,371],{"class":315},[197,1727,728],{"class":354},[197,1729,386],{"class":319},[197,1731,1732,1734,1737],{"class":77,"line":276},[197,1733,345],{"class":315},[197,1735,1736],{"class":202}," pipeline_task",[197,1738,1739],{"class":319},"(self):\n",[197,1741,1742,1744],{"class":77,"line":405},[197,1743,1382],{"class":315},[197,1745,358],{"class":319},[197,1747,1748,1751],{"class":77,"line":420},[197,1749,1750],{"class":315},"        return",[197,1752,1753],{"class":319}," run_pipeline()\n",[197,1755,1756,1758,1761,1764],{"class":77,"line":425},[197,1757,1408],{"class":315},[197,1759,1760],{"class":354}," Exception",[197,1762,1763],{"class":315}," as",[197,1765,1766],{"class":319}," exc:\n",[197,1768,1769,1772,1775,1778,1781,1783,1786,1789,1791,1793],{"class":77,"line":436},[197,1770,1771],{"class":315},"        raise",[197,1773,1774],{"class":354}," self",[197,1776,1777],{"class":319},".retry(",[197,1779,1780],{"class":457},"exc",[197,1782,371],{"class":315},[197,1784,1785],{"class":319},"exc, ",[197,1787,1788],{"class":457},"countdown",[197,1790,371],{"class":315},[197,1792,116],{"class":354},[197,1794,386],{"class":319},[197,1796,1797],{"class":77,"line":441},[197,1798,340],{"emptyLinePlaceholder":339},[197,1800,1801,1804,1806],{"class":77,"line":448},[197,1802,1803],{"class":319},"app.conf.beat_schedule ",[197,1805,371],{"class":315},[197,1807,673],{"class":319},[197,1809,1810,1813],{"class":77,"line":475},[197,1811,1812],{"class":206},"    \"nightly-pipeline\"",[197,1814,1815],{"class":319},": {\n",[197,1817,1818,1821,1823,1826],{"class":77,"line":520},[197,1819,1820],{"class":206},"        \"task\"",[197,1822,497],{"class":319},[197,1824,1825],{"class":206},"\"pipeline_task\"",[197,1827,1147],{"class":319},[197,1829,1830,1833,1836,1839,1841,1844,1846,1849,1851,1853],{"class":77,"line":526},[197,1831,1832],{"class":206},"        \"schedule\"",[197,1834,1835],{"class":319},": crontab(",[197,1837,1838],{"class":457},"hour",[197,1840,371],{"class":315},[197,1842,1843],{"class":354},"2",[197,1845,380],{"class":319},[197,1847,1848],{"class":457},"minute",[197,1850,371],{"class":315},[197,1852,72],{"class":354},[197,1854,731],{"class":319},[197,1856,1857],{"class":77,"line":543},[197,1858,737],{"class":319},[197,1860,1861],{"class":77,"line":548},[197,1862,1863],{"class":319},"}\n",[197,1865,1866,1869,1871,1873,1875,1877,1879],{"class":77,"line":554},[197,1867,1868],{"class":319},"app.conf.timezone ",[197,1870,371],{"class":315},[197,1872,374],{"class":319},[197,1874,1059],{"class":206},[197,1876,380],{"class":319},[197,1878,1064],{"class":206},[197,1880,386],{"class":319},[14,1882,1883,1884,1887,1888,1891,1892,1896],{},"Run ",[194,1885,1886],{},"celery -A app beat"," alongside ",[194,1889,1890],{},"celery -A app worker",". You get free retries, failure isolation per task, and horizontal scaling — at the cost of running Redis and two long-lived processes. There is one operational trap worth stating plainly: run exactly one beat process. If you scale beat to two replicas by accident (easy to do with a naive Kubernetes deployment), every scheduled task fires twice, because beat has no leader election of its own. Keep beat at a single replica and let the workers scale. If Celery feels heavy for the amount of work you actually have, ",[18,1893,1895],{"href":1894},"\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002Fcelery-vs-rq-vs-arq\u002F","Celery vs RQ vs arq"," compares the lighter task queues that still give you a broker-backed schedule.",[163,1898],{},[166,1900,1902],{"id":1901},"step-6-use-a-managed-scheduler-render-railway-github-actions","Step 6 — Use a managed scheduler (Render \u002F Railway \u002F GitHub Actions)",[14,1904,1905],{},"The cheapest option operationally is to let your platform own the clock. A managed cron service spins up your container, runs the entrypoint to completion, and shuts it down — you pay only for the seconds the job runs, and there is no process to keep alive, patch, or monitor for uptime.",[14,1907,1908,1909,1913,1914,1917],{},"Render and Railway both expose a cron job type that just needs a command and a schedule string; they run on the same platform where you already ",[18,1910,1912],{"href":1911},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdeploying-apis-to-render-or-vercel\u002F","deploy your APIs",", so there is no new vendor to onboard. For zero-infrastructure scheduling, GitHub Actions runs your pipeline on a ",[194,1915,1916],{},"schedule"," trigger:",[187,1919,1923],{"className":1920,"code":1921,"language":1922,"meta":192,"style":192},"language-yaml shiki shiki-themes github-light github-dark","name: nightly-pipeline\non:\n  schedule:\n    - cron: \"0 2 * * *\"   # UTC only on GitHub Actions\n  workflow_dispatch:        # manual trigger button\njobs:\n  run:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions\u002Fcheckout@v4\n      - uses: actions\u002Fsetup-python@v5\n        with: { python-version: \"3.11\" }\n      - run: pip install -r requirements.txt\n      - run: python pipeline_entrypoint.py\n        env:\n          SOURCE_API_URL: ${{ secrets.SOURCE_API_URL }}\n          SOURCE_API_KEY: ${{ secrets.SOURCE_API_KEY }}\n","yaml",[194,1924,1925,1936,1943,1950,1964,1975,1982,1989,1999,2006,2019,2030,2049,2061,2072,2079,2089],{"__ignoreMap":192},[197,1926,1927,1931,1933],{"class":77,"line":199},[197,1928,1930],{"class":1929},"s9eBZ","name",[197,1932,497],{"class":319},[197,1934,1935],{"class":206},"nightly-pipeline\n",[197,1937,1938,1941],{"class":77,"line":240},[197,1939,1940],{"class":354},"on",[197,1942,358],{"class":319},[197,1944,1945,1948],{"class":77,"line":246},[197,1946,1947],{"class":1929},"  schedule",[197,1949,358],{"class":319},[197,1951,1952,1955,1957,1959,1961],{"class":77,"line":252},[197,1953,1954],{"class":319},"    - ",[197,1956,872],{"class":1929},[197,1958,497],{"class":319},[197,1960,1090],{"class":206},[197,1962,1963],{"class":444},"   # UTC only on GitHub Actions\n",[197,1965,1966,1969,1972],{"class":77,"line":258},[197,1967,1968],{"class":1929},"  workflow_dispatch",[197,1970,1971],{"class":319},":        ",[197,1973,1974],{"class":444},"# manual trigger button\n",[197,1976,1977,1980],{"class":77,"line":264},[197,1978,1979],{"class":1929},"jobs",[197,1981,358],{"class":319},[197,1983,1984,1987],{"class":77,"line":270},[197,1985,1986],{"class":1929},"  run",[197,1988,358],{"class":319},[197,1990,1991,1994,1996],{"class":77,"line":276},[197,1992,1993],{"class":1929},"    runs-on",[197,1995,497],{"class":319},[197,1997,1998],{"class":206},"ubuntu-latest\n",[197,2000,2001,2004],{"class":77,"line":405},[197,2002,2003],{"class":1929},"    steps",[197,2005,358],{"class":319},[197,2007,2008,2011,2014,2016],{"class":77,"line":420},[197,2009,2010],{"class":319},"      - ",[197,2012,2013],{"class":1929},"uses",[197,2015,497],{"class":319},[197,2017,2018],{"class":206},"actions\u002Fcheckout@v4\n",[197,2020,2021,2023,2025,2027],{"class":77,"line":425},[197,2022,2010],{"class":319},[197,2024,2013],{"class":1929},[197,2026,497],{"class":319},[197,2028,2029],{"class":206},"actions\u002Fsetup-python@v5\n",[197,2031,2032,2035,2038,2041,2043,2046],{"class":77,"line":436},[197,2033,2034],{"class":1929},"        with",[197,2036,2037],{"class":319},": { ",[197,2039,2040],{"class":1929},"python-version",[197,2042,497],{"class":319},[197,2044,2045],{"class":206},"\"3.11\"",[197,2047,2048],{"class":319}," }\n",[197,2050,2051,2053,2056,2058],{"class":77,"line":441},[197,2052,2010],{"class":319},[197,2054,2055],{"class":1929},"run",[197,2057,497],{"class":319},[197,2059,2060],{"class":206},"pip install -r requirements.txt\n",[197,2062,2063,2065,2067,2069],{"class":77,"line":448},[197,2064,2010],{"class":319},[197,2066,2055],{"class":1929},[197,2068,497],{"class":319},[197,2070,2071],{"class":206},"python pipeline_entrypoint.py\n",[197,2073,2074,2077],{"class":77,"line":475},[197,2075,2076],{"class":1929},"        env",[197,2078,358],{"class":319},[197,2080,2081,2084,2086],{"class":77,"line":520},[197,2082,2083],{"class":1929},"          SOURCE_API_URL",[197,2085,497],{"class":319},[197,2087,2088],{"class":206},"${{ secrets.SOURCE_API_URL }}\n",[197,2090,2091,2094,2096],{"class":77,"line":526},[197,2092,2093],{"class":1929},"          SOURCE_API_KEY",[197,2095,497],{"class":319},[197,2097,2098],{"class":206},"${{ secrets.SOURCE_API_KEY }}\n",[14,2100,2101,2102,2105,2106,2108,2109,2113],{},"GitHub Actions cron runs in UTC and can be delayed several minutes under load, so it suits nightly rollups and refreshes, not minute-precise jobs. The ",[194,2103,2104],{},"workflow_dispatch"," line gives you a manual run button in the Actions UI — invaluable for verification and for the \"run it now, I need the report early\" request that inevitably arrives. Two caveats specific to Actions: a ",[194,2107,1916],{}," trigger on a repository with no activity for 60 days is automatically disabled, so a dormant side project quietly stops running; and the secrets you inject are per-repository, so rotating a source API key means updating it in one more place. If your pipeline calls an LLM or another metered vendor on a schedule, keep an eye on the bill — the patterns in ",[18,2110,2112],{"href":2111},"\u002Fautomating-side-hustle-operations-with-apis\u002Fautomating-ai-workflows-with-python-apis\u002Fcontrolling-llm-api-costs-in-production\u002F","controlling LLM API costs in production"," apply directly to a nightly job that quietly makes a thousand paid calls.",[163,2115],{},[166,2117,2119],{"id":2118},"step-7-log-run-status-and-retry-transient-failures","Step 7 — Log run status and retry transient failures",[14,2121,2122],{},"Every scheduled run must emit a machine-readable record of whether it succeeded. Without this you have no way to alert on silent failures, and a scheduled job you cannot see is a scheduled job you cannot trust.",[187,2124,2126],{"className":306,"code":2125,"language":308,"meta":192,"style":192},"import json\nimport logging\nimport sys\n\nlogger = logging.getLogger(\"pipeline\")\nhandler = logging.StreamHandler(sys.stdout)\nlogger.addHandler(handler)\nlogger.setLevel(logging.INFO)\n\ndef run_and_log() -> int:\n    try:\n        result = run_pipeline()\n        logger.info(json.dumps({\"event\": \"pipeline_ok\", **result}))\n        return 0\n    except Exception as exc:\n        logger.error(json.dumps({\"event\": \"pipeline_failed\", \"error\": str(exc)}))\n        return 1\n\nif __name__ == \"__main__\":\n    sys.exit(run_and_log())\n",[194,2127,2128,2135,2142,2148,2152,2166,2176,2181,2191,2195,2209,2215,2224,2245,2252,2262,2286,2293,2297,2309],{"__ignoreMap":192},[197,2129,2130,2132],{"class":77,"line":199},[197,2131,316],{"class":315},[197,2133,2134],{"class":319}," json\n",[197,2136,2137,2139],{"class":77,"line":240},[197,2138,316],{"class":315},[197,2140,2141],{"class":319}," logging\n",[197,2143,2144,2146],{"class":77,"line":246},[197,2145,316],{"class":315},[197,2147,1315],{"class":319},[197,2149,2150],{"class":77,"line":252},[197,2151,340],{"emptyLinePlaceholder":339},[197,2153,2154,2157,2159,2162,2164],{"class":77,"line":258},[197,2155,2156],{"class":319},"logger ",[197,2158,371],{"class":315},[197,2160,2161],{"class":319}," logging.getLogger(",[197,2163,383],{"class":206},[197,2165,386],{"class":319},[197,2167,2168,2171,2173],{"class":77,"line":264},[197,2169,2170],{"class":319},"handler ",[197,2172,371],{"class":315},[197,2174,2175],{"class":319}," logging.StreamHandler(sys.stdout)\n",[197,2177,2178],{"class":77,"line":270},[197,2179,2180],{"class":319},"logger.addHandler(handler)\n",[197,2182,2183,2186,2189],{"class":77,"line":276},[197,2184,2185],{"class":319},"logger.setLevel(logging.",[197,2187,2188],{"class":354},"INFO",[197,2190,386],{"class":319},[197,2192,2193],{"class":77,"line":405},[197,2194,340],{"emptyLinePlaceholder":339},[197,2196,2197,2199,2202,2204,2207],{"class":77,"line":420},[197,2198,345],{"class":315},[197,2200,2201],{"class":202}," run_and_log",[197,2203,351],{"class":319},[197,2205,2206],{"class":354},"int",[197,2208,358],{"class":319},[197,2210,2211,2213],{"class":77,"line":425},[197,2212,1382],{"class":315},[197,2214,358],{"class":319},[197,2216,2217,2220,2222],{"class":77,"line":436},[197,2218,2219],{"class":319},"        result ",[197,2221,371],{"class":315},[197,2223,1753],{"class":319},[197,2225,2226,2229,2232,2234,2237,2239,2242],{"class":77,"line":441},[197,2227,2228],{"class":319},"        logger.info(json.dumps({",[197,2230,2231],{"class":206},"\"event\"",[197,2233,497],{"class":319},[197,2235,2236],{"class":206},"\"pipeline_ok\"",[197,2238,380],{"class":319},[197,2240,2241],{"class":315},"**",[197,2243,2244],{"class":319},"result}))\n",[197,2246,2247,2249],{"class":77,"line":448},[197,2248,1750],{"class":315},[197,2250,2251],{"class":354}," 0\n",[197,2253,2254,2256,2258,2260],{"class":77,"line":475},[197,2255,1408],{"class":315},[197,2257,1760],{"class":354},[197,2259,1763],{"class":315},[197,2261,1766],{"class":319},[197,2263,2264,2267,2269,2271,2274,2276,2279,2281,2283],{"class":77,"line":520},[197,2265,2266],{"class":319},"        logger.error(json.dumps({",[197,2268,2231],{"class":206},[197,2270,497],{"class":319},[197,2272,2273],{"class":206},"\"pipeline_failed\"",[197,2275,380],{"class":319},[197,2277,2278],{"class":206},"\"error\"",[197,2280,497],{"class":319},[197,2282,1356],{"class":354},[197,2284,2285],{"class":319},"(exc)}))\n",[197,2287,2288,2290],{"class":77,"line":526},[197,2289,1750],{"class":315},[197,2291,2292],{"class":354}," 1\n",[197,2294,2295],{"class":77,"line":543},[197,2296,340],{"emptyLinePlaceholder":339},[197,2298,2299,2301,2303,2305,2307],{"class":77,"line":548},[197,2300,1220],{"class":315},[197,2302,1223],{"class":354},[197,2304,1226],{"class":315},[197,2306,1229],{"class":206},[197,2308,358],{"class":319},[197,2310,2311],{"class":77,"line":554},[197,2312,2313],{"class":319},"    sys.exit(run_and_log())\n",[14,2315,2316,2317,2321,2322,2325,2326,2329,2330,2334,2335,2338,2339,2343],{},"Returning a non-zero exit code lets cron, Render, and GitHub Actions all mark the run as failed and trigger their own alerting — an emailed cron failure or a red Actions run is free monitoring you do not have to build. For richer pipelines, graduate to ",[18,2318,2320],{"href":2319},"\u002Fscaling-and-operating-production-python-apis\u002Fmonitoring-and-logging-python-apis\u002Fstructured-logging-with-structlog\u002F","structured logging with structlog"," so every run emits a consistent JSON event you can query later. There is one distinction worth drawing: a ",[27,2323,2324],{},"transient"," failure (the source API returned a 503, the network blipped) deserves an in-run retry with backoff, while a ",[27,2327,2328],{},"permanent"," failure (bad credentials, a schema change) should fail loudly and stop. Do not paper over a permanent failure with retries; you will just burn three attempts and still fail, only later. Wrap the extract step with ",[18,2331,2333],{"href":2332},"\u002Fgetting-started-with-python-apis-for-builders\u002Fmaking-http-requests-with-requests-library\u002Fretrying-failed-http-requests-with-tenacity\u002F","tenacity retries"," tuned to retry on connection errors and 5xx responses but not on a 401, and let the outer ",[194,2336,2337],{},"run_and_log"," catch what tenacity gives up on. Before you ship, cover the pipeline with a test that mocks the source API and asserts the upsert is idempotent — the approach in ",[18,2340,2342],{"href":2341},"\u002Fscaling-and-operating-production-python-apis\u002Ftesting-python-apis-with-pytest\u002F","testing Python APIs with pytest"," works for a scheduled job just as well as for a request handler.",[163,2345],{},[166,2347,2349],{"id":2348},"configuration-reference","Configuration reference",[2351,2352,2353,2369],"table",{},[2354,2355,2356],"thead",{},[2357,2358,2359,2363,2366],"tr",{},[2360,2361,2362],"th",{},"Env var",[2360,2364,2365],{},"Default",[2360,2367,2368],{},"Production recommendation",[2370,2371,2372,2392,2411,2426,2440,2453],"tbody",{},[2357,2373,2374,2380,2385],{},[2375,2376,2377],"td",{},[194,2378,2379],{},"CRON_SCHEDULE",[2375,2381,2382],{},[194,2383,2384],{},"0 2 * * *",[2375,2386,2387,2388,2391],{},"Off-peak hours; avoid ",[194,2389,2390],{},"0 0 * * *"," rush. Quote it in YAML.",[2357,2393,2394,2399,2404],{},[2375,2395,2396],{},[194,2397,2398],{},"PIPELINE_TZ",[2375,2400,2401],{},[194,2402,2403],{},"UTC",[2375,2405,2406,2407,2410],{},"Pin a real zone (e.g. ",[194,2408,2409],{},"Europe\u002FBrussels","); GitHub Actions ignores it (UTC only).",[2357,2412,2413,2418,2423],{},[2375,2414,2415],{},[194,2416,2417],{},"LOCK_PATH",[2375,2419,2420],{},[194,2421,2422],{},"\u002Ftmp\u002Fpipeline.lock",[2375,2424,2425],{},"A durable path per pipeline; one lock file per distinct job.",[2357,2427,2428,2433,2437],{},[2375,2429,2430],{},[194,2431,2432],{},"LOCK_TTL_SECONDS",[2375,2434,2435],{},[194,2436,1189],{},[2375,2438,2439],{},"Slightly above your worst-case run time, so a crashed run self-clears.",[2357,2441,2442,2447,2450],{},[2375,2443,2444],{},[194,2445,2446],{},"CELERY_BROKER_URL",[2375,2448,2449],{},"—",[2375,2451,2452],{},"A managed Redis URL; never the local default in production.",[2357,2454,2455,2459,2461],{},[2375,2456,2457],{},[194,2458,1550],{},[2375,2460,2449],{},[2375,2462,2463],{},"Inject from a secret manager; never commit.",[163,2465],{},[166,2467,2469],{"id":2468},"gotchas-failure-modes","Gotchas & failure modes",[2471,2472,2473,2487,2496,2508,2514,2526],"ol",{},[2474,2475,2476,2480,2481,2483,2484,2486],"li",{},[2477,2478,2479],"strong",{},"Overlapping runs."," A slow nightly job still running when the next trigger fires gives you two copies fighting over the same data. APScheduler's ",[194,2482,1242],{},", the OS-level ",[194,2485,1546],{},", or Celery's broker dedup each prevent this — pick one and actually wire it in. The idempotent upsert is your seatbelt if one slips through.",[2474,2488,2489,2492,2493,2495],{},[2477,2490,2491],{},"Timezone and DST."," ",[194,2494,2384],{}," runs twice on the fall-back night and skips on spring-forward in a DST zone. Schedule in UTC, or pin an explicit IANA timezone in your scheduler so it handles the transition. Do not hand-roll offsets; they rot at the next DST boundary.",[2474,2497,2498,2501,2502,2504,2505,2507],{},[2477,2499,2500],{},"Missed runs on restart."," If a deploy restarts your container at 2:00 AM, an in-process APScheduler job is simply gone. Use ",[194,2503,1246],{}," + ",[194,2506,1273],{},", or move the clock to a managed cron that does not depend on your process being up.",[2474,2509,2510,2513],{},[2477,2511,2512],{},"No alerting on failure."," A job that fails silently for a week is worse than no job — you make decisions on stale data and never know. Always return a non-zero exit code and let the platform alert, or post to a webhook on the failure branch.",[2474,2515,2516,2519,2520,2522,2523,2525],{},[2477,2517,2518],{},"Long jobs blocking everything."," A multi-minute sync inside an ",[194,2521,927],{}," blocks the event loop unless you offload it with ",[194,2524,1254],{},". If jobs routinely run long, that is the signal to move to Celery workers, not to add more threads.",[2474,2527,2528,2531],{},[2477,2529,2530],{},"Duplicate beat processes."," Two Celery beat replicas fire every task twice. Keep beat at a single replica; scale the workers, never the scheduler.",[163,2533],{},[166,2535,2537],{"id":2536},"verification","Verification",[14,2539,2540],{},"Confirm the schedule without waiting until 2:00 AM. Trigger the pipeline manually and inspect the structured log:",[187,2542,2544],{"className":189,"code":2543,"language":191,"meta":192,"style":192},"PIPELINE_NAME=test python pipeline_entrypoint.py\n# {\"event\": \"pipeline_ok\", \"pipeline\": \"test\", \"fetched\": 12, \"loaded\": 12, \"duration_s\": 0.84}\necho \"exit code: $?\"   # expect 0\n",[194,2545,2546,2562,2567],{"__ignoreMap":192},[197,2547,2548,2551,2553,2556,2559],{"class":77,"line":199},[197,2549,2550],{"class":319},"PIPELINE_NAME",[197,2552,371],{"class":315},[197,2554,2555],{"class":206},"test",[197,2557,2558],{"class":202}," python",[197,2560,2561],{"class":206}," pipeline_entrypoint.py\n",[197,2563,2564],{"class":77,"line":240},[197,2565,2566],{"class":444},"# {\"event\": \"pipeline_ok\", \"pipeline\": \"test\", \"fetched\": 12, \"loaded\": 12, \"duration_s\": 0.84}\n",[197,2568,2569,2572,2575,2578,2580],{"class":77,"line":246},[197,2570,2571],{"class":354},"echo",[197,2573,2574],{"class":206}," \"exit code: ",[197,2576,2577],{"class":354},"$?",[197,2579,514],{"class":206},[197,2581,2582],{"class":444},"   # expect 0\n",[14,2584,2585],{},"For APScheduler, log the next fire time on startup so you can eyeball that the cron expression parsed the way you meant:",[187,2587,2589],{"className":306,"code":2588,"language":308,"meta":192,"style":192},"job = scheduler.get_job(\"nightly-pipeline\")\nprint(\"next run:\", job.next_run_time)\n",[194,2590,2591,2605],{"__ignoreMap":192},[197,2592,2593,2596,2598,2601,2603],{"class":77,"line":199},[197,2594,2595],{"class":319},"job ",[197,2597,371],{"class":315},[197,2599,2600],{"class":319}," scheduler.get_job(",[197,2602,1144],{"class":206},[197,2604,386],{"class":319},[197,2606,2607,2610,2612,2615],{"class":77,"line":240},[197,2608,2609],{"class":354},"print",[197,2611,1421],{"class":319},[197,2613,2614],{"class":206},"\"next run:\"",[197,2616,2617],{"class":319},", job.next_run_time)\n",[14,2619,2620,2621,2624,2625,2627,2628,2631],{},"On GitHub Actions, click ",[2477,2622,2623],{},"Run workflow"," (the ",[194,2626,2104],{}," button) and confirm the run goes green. On Render or Railway, check the cron job's run history for a successful exit. The best verification is a deliberate failure: point ",[194,2629,2630],{},"SOURCE_API_URL"," at a URL that returns 500, run once, and confirm you actually get the alert. An untested alert path is the reason silent failures survive for a week.",[163,2633],{},[166,2635,2637],{"id":2636},"cost-performance-note","Cost & performance note",[14,2639,2640],{},"The trade-off is always-on versus pay-per-run, and for a side hustle the numbers are not close. An always-on container with APScheduler costs the same whether it runs the job once a night or never — typically $5–7\u002Fmonth minimum on a small instance. Add Celery beat and you are running the app container plus a broker, so realistic all-in is $10–13\u002Fmonth before you have processed a single order. A managed serverless cron bills only for the seconds the job runs: a nightly five-minute pipeline on Render cron is roughly 150 minutes of compute a month, which lands in the low tens of cents; on GitHub Actions for a private repo it is a rounding error against the free minutes, and free on a public repo.",[32,2642,41,2646,41,2649,41,2652,41,2654,41,2658,41,2661,41,2666,41,2671,41,2676,41,2679,41,2683,41,2688,41,2692,41,2696,41,2701,41,2705,41,2707,41,2711],{"viewBox":1557,"role":35,"ariaLabelledBy":2643,"xmlns":39,"style":40},[2644,2645],"cron-cost-t","cron-cost-d",[43,2647,2648],{"id":2644},"Approximate monthly cost of four scheduling options",[47,2650,2651],{"id":2645},"Horizontal bars comparing a Celery beat stack near thirteen dollars, an APScheduler container near seven dollars, Render cron near thirty cents, and GitHub Actions near zero for one nightly job.",[70,2653],{"x":72,"y":72,"width":73,"height":1569,"fill":75},[100,2655,2657],{"x":862,"y":103,"fill":123,"style":2656},"text-anchor:start;font-size:13;font-family:var(--font-sans);","Monthly cost, one nightly 5-minute job (approx.)",[77,2659],{"x1":863,"y1":2660,"x2":863,"y2":128,"stroke":82,"style":1584},"50",[100,2662,2665],{"x":839,"y":2663,"fill":68,"style":2664},"76","text-anchor:end;font-size:12;font-family:var(--font-sans);","Celery beat",[70,2667],{"x":863,"y":2668,"width":2669,"height":103,"rx":2670,"fill":143},"62","468","4",[100,2672,2675],{"x":2673,"y":2674,"fill":123,"style":2664},"676","80","~$13",[100,2677,887],{"x":839,"y":2678,"fill":68,"style":2664},"122",[70,2680],{"x":863,"y":2681,"width":2682,"height":103,"rx":2670,"fill":89},"108","252",[100,2684,2687],{"x":2685,"y":2686,"fill":123,"style":1572},"460","126","~$7",[100,2689,2691],{"x":839,"y":2690,"fill":68,"style":2664},"168","Render cron",[70,2693],{"x":863,"y":2694,"width":2695,"height":103,"rx":2670,"fill":133},"154","14",[100,2697,2700],{"x":2698,"y":2699,"fill":123,"style":1572},"222","172","~$0.30",[100,2702,2704],{"x":839,"y":2703,"fill":68,"style":2664},"214","GitHub Actions",[70,2706],{"x":863,"y":863,"width":88,"height":103,"rx":728,"fill":118},[100,2708,2710],{"x":2703,"y":2709,"fill":123,"style":1572},"218","~$0",[100,2712,2714],{"x":102,"y":2713,"fill":68,"style":1590},"248","Bars scaled to dollars; serverless options round to cents at side-hustle volume.",[14,2716,2717,2718,2722],{},"So for a side hustle running a handful of scheduled jobs, serverless cron wins decisively — the entire scheduling line item disappears into the noise of your other costs. Reach for an always-on scheduler only when you also need the process online for live traffic anyway (then APScheduler rides for free on infrastructure you already pay for), or when jobs run frequently enough — every few minutes — that repeated container cold starts cost more time and money than keeping one warm. As your automation grows past a couple of jobs into a real network of triggers and actions, the calculus shifts again toward ",[18,2719,2721],{"href":2720},"\u002Fautomating-side-hustle-operations-with-apis\u002Fbuilding-zapier-alternatives-with-python\u002F","building Zapier alternatives with Python",", where a single always-on process orchestrating many workflows finally earns its monthly cost.",[163,2724],{},[166,2726,2728],{"id":2727},"faq","FAQ",[14,2730,2731,2734],{},[2477,2732,2733],{},"Should I use cron or APScheduler for a side-hustle pipeline?","\nIf you do not already run an always-on process, use OS cron or a managed cron service — it is free, survives restarts, and needs no broker. Choose APScheduler only when you already have a long-running app where adding an in-process schedule avoids spinning up separate infrastructure, so the scheduler rides for free on compute you are already paying for.",[14,2736,2737,2740],{},[2477,2738,2739],{},"How much does it cost to run a nightly pipeline?","\nOn managed serverless cron, cents or nothing: a five-minute nightly job is about 150 minutes of compute a month, which lands in the low tens of cents on Render and free on GitHub Actions for a public repo. An always-on APScheduler container is $5–7\u002Fmonth whether it runs or not, and a Celery beat stack with a broker is $10–13. At side-hustle volume the serverless options make the scheduling line item vanish.",[14,2742,2743,2746,2747,2749,2750,2752],{},[2477,2744,2745],{},"How do I stop a scheduled job from running twice at once?","\nUse one overlap guard end to end: ",[194,2748,1242],{}," in APScheduler, a non-blocking ",[194,2751,1546],{}," in an OS-cron entrypoint, or rely on Celery's broker so a single task is dequeued once. The bigger insurance is making the pipeline idempotent with an upsert so a rare double-run cannot corrupt data anyway — that lets you use the cheapest scheduler without fear.",[14,2754,2755,2758,2759,2761,2762,2764],{},[2477,2756,2757],{},"Why did my nightly job not run after a deploy?","\nIn-process schedulers lose their state on restart, so a job scheduled for a moment the container was down is simply missed. Set ",[194,2760,1246],{}," with a ",[194,2763,1273],{},", or move scheduling to a managed cron that triggers independently of your app's uptime. On GitHub Actions specifically, a scheduled workflow is auto-disabled after 60 days of repository inactivity, which silently stops a dormant side project.",[14,2766,2767,2770,2771,2773],{},[2477,2768,2769],{},"When is it worth migrating from cron to Celery beat?","\nWhen you already run Celery workers for other work, or when a single job needs to coordinate across multiple machines that a local ",[194,2772,1546],{}," cannot cover. Until then, Celery beat adds a broker, a second process, and a duplicate-beat footgun for no gain. Migrate for a concrete coordination or retry need, not because it feels more \"production-grade.\"",[163,2775],{},[166,2777,2779],{"id":2778},"related","Related",[14,2781,2782],{},[2477,2783,2784],{},"Same track:",[2786,2787,2788,2793,2799,2805],"ul",{},[2474,2789,2790,2792],{},[18,2791,1278],{"href":1277}," — the durability and complexity trade-off between the two in-code schedulers.",[2474,2794,2795,2798],{},[18,2796,2797],{"href":175},"Sync Shopify Orders to Google Sheets via API"," — the pipeline body this guide schedules.",[2474,2800,2801,2804],{},[18,2802,2803],{"href":2720},"Building Zapier Alternatives with Python"," — where scheduling grows into a full workflow engine.",[2474,2806,2807,2809],{},[18,2808,21],{"href":20}," — the section overview for all of this.",[14,2811,2812],{},[2477,2813,2814],{},"Other tracks:",[2786,2816,2817,2823,2829],{},[2474,2818,2819,2822],{},[18,2820,2821],{"href":913},"Running Background Jobs with Celery"," — the worker layer Celery beat enqueues onto.",[2474,2824,2825,2828],{},[18,2826,2827],{"href":2332},"Retrying Failed HTTP Requests with tenacity"," — backoff for the transient failures a scheduled extract will hit.",[2474,2830,2831,2834],{},[18,2832,2833],{"href":2319},"Structured Logging with structlog"," — turn run-status logs into something you can query and alert on.",[2836,2837,2838],"style",{},"html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html pre.shiki code .s9eBZ, html code.shiki .s9eBZ{--shiki-default:#22863A;--shiki-dark:#85E89D}",{"title":192,"searchDepth":240,"depth":240,"links":2840},[2841,2842,2843,2844,2845,2846,2847,2848,2849,2850,2851,2852,2853,2854],{"id":168,"depth":240,"text":169},{"id":295,"depth":240,"text":296},{"id":773,"depth":240,"text":774},{"id":920,"depth":240,"text":921},{"id":1284,"depth":240,"text":1285},{"id":1621,"depth":240,"text":1622},{"id":1901,"depth":240,"text":1902},{"id":2118,"depth":240,"text":2119},{"id":2348,"depth":240,"text":2349},{"id":2468,"depth":240,"text":2469},{"id":2536,"depth":240,"text":2537},{"id":2636,"depth":240,"text":2637},{"id":2727,"depth":240,"text":2728},{"id":2778,"depth":240,"text":2779},"Schedule recurring Python data pipelines — nightly syncs, report rollups, scraping refreshes — with cron, APScheduler, Celery beat, or managed serverless cron.","md",{"pageTitle":2858,"datePublished":2859,"dateModified":2860},"Schedule Python Data Pipelines with Cron","2026-06-18","2026-07-23","\u002Fautomating-side-hustle-operations-with-apis\u002Fscheduling-data-pipelines-with-cron",{"title":5,"description":2855},"automating-side-hustle-operations-with-apis\u002Fscheduling-data-pipelines-with-cron\u002Findex","VqGxudOJAEbDCnCRu6gBNfcoHEWOuer8DP1oZ_hrRk0",{"@context":2866,"@type":2867,"mainEntity":2868},"https:\u002F\u002Fschema.org","FAQPage",[2869,2874,2877,2880,2883],{"@type":2870,"name":2733,"acceptedAnswer":2871},"Question",{"@type":2872,"text":2873},"Answer","If you do not already run an always-on process, use OS cron or a managed cron service — it is free, survives restarts, and needs no broker. Choose APScheduler only when you already have a long-running app where adding an in-process schedule avoids spinning up separate infrastructure, so the scheduler rides for free on compute you are already paying for.",{"@type":2870,"name":2739,"acceptedAnswer":2875},{"@type":2872,"text":2876},"On managed serverless cron, cents or nothing: a five-minute nightly job is about 150 minutes of compute a month, which lands in the low tens of cents on Render and free on GitHub Actions for a public repo. An always-on APScheduler container is $5–7\u002Fmonth whether it runs or not, and a Celery beat stack with a broker is $10–13. At side-hustle volume the serverless options make the scheduling line item vanish.",{"@type":2870,"name":2745,"acceptedAnswer":2878},{"@type":2872,"text":2879},"Use one overlap guard end to end: max_instances=1 in APScheduler, a non-blocking flock in an OS-cron entrypoint, or rely on Celery's broker so a single task is dequeued once. The bigger insurance is making the pipeline idempotent with an upsert so a rare double-run cannot corrupt data anyway — that lets you use the cheapest scheduler without fear.",{"@type":2870,"name":2757,"acceptedAnswer":2881},{"@type":2872,"text":2882},"In-process schedulers lose their state on restart, so a job scheduled for a moment the container was down is simply missed. Set coalesce=True with a misfire_grace_time, or move scheduling to a managed cron that triggers independently of your app's uptime. On GitHub Actions specifically, a scheduled workflow is auto-disabled after 60 days of repository inactivity, which silently stops a dormant side project.",{"@type":2870,"name":2769,"acceptedAnswer":2884},{"@type":2872,"text":2885},"When you already run Celery workers for other work, or when a single job needs to coordinate across multiple machines that a local flock cannot cover. Until then, Celery beat adds a broker, a second process, and a duplicate-beat footgun for no gain. Migrate for a concrete coordination or retry need, not because it feels more \"production-grade.\" ---",1784887027769]