[{"data":1,"prerenderedAt":1330},["ShallowReactive",2],{"page-\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002Fcelery-vs-rq-vs-arq\u002F":3,"faq-schema-\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002Fcelery-vs-rq-vs-arq\u002F":1309},{"id":4,"title":5,"body":6,"description":1298,"extension":1299,"meta":1300,"navigation":375,"path":1305,"seo":1306,"stem":1307,"__hash__":1308},"content\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002Fcelery-vs-rq-vs-arq\u002Findex.md","Celery vs RQ vs arq: Choosing a Python Background Job Queue",{"type":7,"value":8,"toc":1286},"minimark",[9,13,23,31,133,138,276,287,294,298,314,654,661,876,883,978,982,985,996,1071,1083,1087,1101,1107,1122,1127,1135,1139,1167,1171,1183,1187,1193,1199,1209,1215,1221,1225,1230,1258,1263,1282],[10,11,5],"h1",{"id":12},"celery-vs-rq-vs-arq-choosing-a-python-background-job-queue",[14,15,16,17,22],"p",{},"Once you decide to offload slow work off your request path, the next question is which queue to run it on. The three names that come up are Celery, RQ, and arq — and they sit at very different points on the spectrum from \"battle-tested but heavy\" to \"tiny and async-native.\" Getting this choice wrong is expensive: you either over-build a distributed task system for a job that fires three times a day, or you bolt a synchronous worker onto an async API and quietly burn threads and money on every request. This comparison helps you pick before you wire anything in. Part of the ",[18,19,21],"a",{"href":20},"\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002F","Running Background Jobs with Celery"," guide.",[14,24,25,26,30],{},"The deciding factor for most builders is two things at once: whether your API is async (FastAPI on ",[27,28,29],"code",{},"asyncio",") and how much operational machinery you are willing to pay for and keep alive. Everything below flows from those two axes. If you are running an async stack — and most people shipping a first or second commercial API today are — the answer skews hard toward arq, and the rest of this page is about knowing when it does not.",[32,33,41,42,41,46,41,50,41,57,41,65,41,68,41,76,41,80,41,86,41,91,41,99,41,106,41,110,41,113,41,118,41,122,41,126,41,130],"svg",{"viewBox":34,"role":35,"ariaLabelledBy":36,"xmlns":39,"style":40},"0 0 720 280","img",[37,38],"cvrva-matrix-t","cvrva-matrix-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;height:auto;margin:1.5rem 0;font-family:var(--font-sans);","\n  ",[43,44,45],"title",{"id":37},"Comparison matrix of Celery, RQ, and arq",[47,48,49],"desc",{"id":38},"A matrix plotting Celery, RQ, and arq against async-native support on the horizontal axis and operational complexity on the vertical axis. Celery sits high-complexity and sync, RQ low-complexity and sync, arq low-complexity and async-native.",[51,52],"rect",{"x":53,"y":53,"width":54,"height":55,"fill":56},"0","720","280","var(--c-surface)",[58,59],"line",{"x1":60,"y1":61,"x2":60,"y2":62,"stroke":63,"style":64},"80","40","240","var(--c-border)","stroke-width:2;",[58,66],{"x1":60,"y1":62,"x2":67,"y2":62,"stroke":63,"style":64},"690",[69,70,75],"text",{"x":71,"y":72,"fill":73,"style":74},"50","44","var(--c-text-muted)","text-anchor:middle;font-size:11;font-family:var(--font-sans);","high",[69,77,79],{"x":71,"y":78,"fill":73,"style":74},"236","low",[69,81,85],{"x":61,"y":82,"fill":73,"transform":83,"style":84},"150","rotate(-90 40 150)","text-anchor:middle;font-size:12;font-family:var(--font-sans);","Ops complexity",[69,87,90],{"x":88,"y":89,"fill":73,"style":84},"385","266","Async-native →",[51,92],{"x":93,"y":94,"width":82,"height":95,"rx":96,"fill":97,"stroke":98,"style":64},"120","60","56","10","var(--c-surface-alt)","var(--c-blue)",[69,100,105],{"x":101,"y":102,"fill":103,"style":104},"195","84","var(--c-text)","text-anchor:middle;font-size:14;font-family:var(--font-sans);","Celery",[69,107,109],{"x":101,"y":108,"fill":73,"style":74},"102","heavy, sync core",[51,111],{"x":82,"y":82,"width":82,"height":95,"rx":96,"fill":97,"stroke":112,"style":64},"var(--c-teal)",[69,114,117],{"x":115,"y":116,"fill":103,"style":104},"225","174","RQ",[69,119,121],{"x":115,"y":120,"fill":73,"style":74},"192","simple, sync",[51,123],{"x":124,"y":82,"width":82,"height":95,"rx":96,"fill":97,"stroke":125,"style":64},"500","var(--c-coral)",[69,127,129],{"x":128,"y":116,"fill":103,"style":104},"575","arq",[69,131,132],{"x":128,"y":120,"fill":73,"style":74},"async-native",[134,135,137],"h2",{"id":136},"side-by-side-comparison","Side-by-side comparison",[139,140,141,156],"table",{},[142,143,144],"thead",{},[145,146,147,150,152,154],"tr",{},[148,149],"th",{},[148,151,105],{},[148,153,117],{},[148,155,129],{},[157,158,159,180,195,210,229,245,260],"tbody",{},[145,160,161,168,171,174],{},[162,163,164],"td",{},[165,166,167],"strong",{},"Async-native?",[162,169,170],{},"No (sync workers)",[162,172,173],{},"No (sync only)",[162,175,176,177,179],{},"Yes (",[27,178,29],{},")",[145,181,182,187,190,193],{},[162,183,184],{},[165,185,186],{},"Broker",[162,188,189],{},"Redis, RabbitMQ, SQS",[162,191,192],{},"Redis only",[162,194,192],{},[145,196,197,202,205,208],{},[162,198,199],{},[165,200,201],{},"Complexity",[162,203,204],{},"High",[162,206,207],{},"Low",[162,209,207],{},[145,211,212,217,220,226],{},[162,213,214],{},[165,215,216],{},"Scheduling",[162,218,219],{},"Celery beat",[162,221,222,225],{},[27,223,224],{},"rq-scheduler"," add-on",[162,227,228],{},"Built-in cron",[145,230,231,236,239,242],{},[162,232,233],{},[165,234,235],{},"Ops overhead",[162,237,238],{},"Beat + workers + Flower",[162,240,241],{},"Worker + optional dash",[162,243,244],{},"Single worker",[145,246,247,252,255,258],{},[162,248,249],{},[165,250,251],{},"Result backend",[162,253,254],{},"Pluggable",[162,256,257],{},"Redis",[162,259,257],{},[145,261,262,267,270,273],{},[162,263,264],{},[165,265,266],{},"Best for",[162,268,269],{},"Complex multi-broker systems",[162,271,272],{},"Simple sync apps",[162,274,275],{},"Async FastAPI services",[14,277,278,279,283,284,286],{},"The headline split is about where your task code runs. Celery and RQ execute synchronous workers, so an async HTTP call inside a task has to run in a thread pool or a per-task event loop you spin up by hand. arq runs your task ",[280,281,282],"em",{},"as a coroutine"," on the same ",[27,285,29],{}," machinery FastAPI already uses, which means it shares your async HTTP client, async database drivers, and connection pools with zero adaptation. That single fact — coroutine versus blocking function — cascades into throughput, cost, and how much code you write to keep the two worlds in sync.",[14,288,289,290,293],{},"Broker flexibility is the other real dividing line. RQ and arq both require ",[18,291,257],{"href":292},"\u002Fscaling-and-operating-production-python-apis\u002Fcaching-python-api-responses-with-redis\u002F"," and only Redis, which is a feature, not a limitation, for most builders — you almost certainly already run Redis for caching, so the queue costs you nothing extra to stand up. Celery is the only one that speaks RabbitMQ and Amazon SQS. If your architecture already commits to one of those, or you need SQS's managed durability so you never operate a broker yourself, that alone can decide the question before any code is written.",[134,295,297],{"id":296},"contrasting-an-arq-task-with-a-celery-task","Contrasting an arq task with a Celery task",[14,299,300,301,304,305,308,309,313],{},"The same job — POST a payload to a webhook — looks meaningfully different in each. Here is arq, where the task is ",[27,302,303],{},"async def"," and uses a shared async HTTP client directly. Notice the ",[27,306,307],{},"ctx"," dict: arq threads a per-worker context through every job, which is where you stash long-lived resources like a pooled ",[18,310,312],{"href":311},"\u002Fgetting-started-with-python-apis-for-builders\u002Fmaking-http-requests-with-requests-library\u002Fhttpx-vs-requests-for-async\u002F","httpx"," client or a database engine.",[315,316,321],"pre",{"className":317,"code":318,"language":319,"meta":320,"style":320},"language-python shiki shiki-themes github-light github-dark","import os\nimport httpx\nfrom arq import create_pool\nfrom arq.connections import RedisSettings\n\nasync def deliver_webhook(ctx, url: str, payload: dict) -> int:\n    # ctx[\"session\"] is a shared httpx.AsyncClient created once on startup\n    resp = await ctx[\"session\"].post(url, json=payload, timeout=10.0)\n    resp.raise_for_status()\n    return resp.status_code\n\nasync def startup(ctx):\n    ctx[\"session\"] = httpx.AsyncClient()\n\nasync def shutdown(ctx):\n    await ctx[\"session\"].aclose()\n\nclass WorkerSettings:\n    functions = [deliver_webhook]\n    on_startup = startup\n    on_shutdown = shutdown\n    max_jobs = int(os.getenv(\"ARQ_MAX_JOBS\", \"100\"))\n    redis_settings = RedisSettings.from_dsn(os.getenv(\"REDIS_URL\"))\n\n# Enqueue from a FastAPI route:\n# pool = await create_pool(WorkerSettings.redis_settings)\n# await pool.enqueue_job(\"deliver_webhook\", url, payload)\n","python","",[27,322,323,335,343,357,370,377,412,419,461,467,476,481,494,510,515,527,540,545,556,567,578,589,615,631,636,642,648],{"__ignoreMap":320},[324,325,327,331],"span",{"class":58,"line":326},1,[324,328,330],{"class":329},"szBVR","import",[324,332,334],{"class":333},"sVt8B"," os\n",[324,336,338,340],{"class":58,"line":337},2,[324,339,330],{"class":329},[324,341,342],{"class":333}," httpx\n",[324,344,346,349,352,354],{"class":58,"line":345},3,[324,347,348],{"class":329},"from",[324,350,351],{"class":333}," arq ",[324,353,330],{"class":329},[324,355,356],{"class":333}," create_pool\n",[324,358,360,362,365,367],{"class":58,"line":359},4,[324,361,348],{"class":329},[324,363,364],{"class":333}," arq.connections ",[324,366,330],{"class":329},[324,368,369],{"class":333}," RedisSettings\n",[324,371,373],{"class":58,"line":372},5,[324,374,376],{"emptyLinePlaceholder":375},true,"\n",[324,378,380,383,386,390,393,397,400,403,406,409],{"class":58,"line":379},6,[324,381,382],{"class":329},"async",[324,384,385],{"class":329}," def",[324,387,389],{"class":388},"sScJk"," deliver_webhook",[324,391,392],{"class":333},"(ctx, url: ",[324,394,396],{"class":395},"sj4cs","str",[324,398,399],{"class":333},", payload: ",[324,401,402],{"class":395},"dict",[324,404,405],{"class":333},") -> ",[324,407,408],{"class":395},"int",[324,410,411],{"class":333},":\n",[324,413,415],{"class":58,"line":414},7,[324,416,418],{"class":417},"sJ8bj","    # ctx[\"session\"] is a shared httpx.AsyncClient created once on startup\n",[324,420,422,425,428,431,434,438,441,445,447,450,453,455,458],{"class":58,"line":421},8,[324,423,424],{"class":333},"    resp ",[324,426,427],{"class":329},"=",[324,429,430],{"class":329}," await",[324,432,433],{"class":333}," ctx[",[324,435,437],{"class":436},"sZZnC","\"session\"",[324,439,440],{"class":333},"].post(url, ",[324,442,444],{"class":443},"s4XuR","json",[324,446,427],{"class":329},[324,448,449],{"class":333},"payload, ",[324,451,452],{"class":443},"timeout",[324,454,427],{"class":329},[324,456,457],{"class":395},"10.0",[324,459,460],{"class":333},")\n",[324,462,464],{"class":58,"line":463},9,[324,465,466],{"class":333},"    resp.raise_for_status()\n",[324,468,470,473],{"class":58,"line":469},10,[324,471,472],{"class":329},"    return",[324,474,475],{"class":333}," resp.status_code\n",[324,477,479],{"class":58,"line":478},11,[324,480,376],{"emptyLinePlaceholder":375},[324,482,484,486,488,491],{"class":58,"line":483},12,[324,485,382],{"class":329},[324,487,385],{"class":329},[324,489,490],{"class":388}," startup",[324,492,493],{"class":333},"(ctx):\n",[324,495,497,500,502,505,507],{"class":58,"line":496},13,[324,498,499],{"class":333},"    ctx[",[324,501,437],{"class":436},[324,503,504],{"class":333},"] ",[324,506,427],{"class":329},[324,508,509],{"class":333}," httpx.AsyncClient()\n",[324,511,513],{"class":58,"line":512},14,[324,514,376],{"emptyLinePlaceholder":375},[324,516,518,520,522,525],{"class":58,"line":517},15,[324,519,382],{"class":329},[324,521,385],{"class":329},[324,523,524],{"class":388}," shutdown",[324,526,493],{"class":333},[324,528,530,533,535,537],{"class":58,"line":529},16,[324,531,532],{"class":329},"    await",[324,534,433],{"class":333},[324,536,437],{"class":436},[324,538,539],{"class":333},"].aclose()\n",[324,541,543],{"class":58,"line":542},17,[324,544,376],{"emptyLinePlaceholder":375},[324,546,548,551,554],{"class":58,"line":547},18,[324,549,550],{"class":329},"class",[324,552,553],{"class":388}," WorkerSettings",[324,555,411],{"class":333},[324,557,559,562,564],{"class":58,"line":558},19,[324,560,561],{"class":333},"    functions ",[324,563,427],{"class":329},[324,565,566],{"class":333}," [deliver_webhook]\n",[324,568,570,573,575],{"class":58,"line":569},20,[324,571,572],{"class":333},"    on_startup ",[324,574,427],{"class":329},[324,576,577],{"class":333}," startup\n",[324,579,581,584,586],{"class":58,"line":580},21,[324,582,583],{"class":333},"    on_shutdown ",[324,585,427],{"class":329},[324,587,588],{"class":333}," shutdown\n",[324,590,592,595,597,600,603,606,609,612],{"class":58,"line":591},22,[324,593,594],{"class":333},"    max_jobs ",[324,596,427],{"class":329},[324,598,599],{"class":395}," int",[324,601,602],{"class":333},"(os.getenv(",[324,604,605],{"class":436},"\"ARQ_MAX_JOBS\"",[324,607,608],{"class":333},", ",[324,610,611],{"class":436},"\"100\"",[324,613,614],{"class":333},"))\n",[324,616,618,621,623,626,629],{"class":58,"line":617},23,[324,619,620],{"class":333},"    redis_settings ",[324,622,427],{"class":329},[324,624,625],{"class":333}," RedisSettings.from_dsn(os.getenv(",[324,627,628],{"class":436},"\"REDIS_URL\"",[324,630,614],{"class":333},[324,632,634],{"class":58,"line":633},24,[324,635,376],{"emptyLinePlaceholder":375},[324,637,639],{"class":58,"line":638},25,[324,640,641],{"class":417},"# Enqueue from a FastAPI route:\n",[324,643,645],{"class":58,"line":644},26,[324,646,647],{"class":417},"# pool = await create_pool(WorkerSettings.redis_settings)\n",[324,649,651],{"class":58,"line":650},27,[324,652,653],{"class":417},"# await pool.enqueue_job(\"deliver_webhook\", url, payload)\n",[14,655,656,657,660],{},"And the Celery equivalent, where the task is a regular ",[27,658,659],{},"def"," and you make a synchronous call. To get the same resilience you lean on Celery's built-in retry machinery rather than an external library:",[315,662,664],{"className":317,"code":663,"language":319,"meta":320,"style":320},"import os\nimport httpx\nfrom celery import Celery\n\ncelery_app = Celery(\"jobs\", broker=os.getenv(\"CELERY_BROKER_URL\"))\n\n@celery_app.task(name=\"jobs.deliver_webhook\", bind=True, max_retries=5)\ndef deliver_webhook(self, url: str, payload: dict) -> int:\n    try:\n        resp = httpx.post(url, json=payload, timeout=10.0)\n        resp.raise_for_status()\n    except httpx.HTTPError as exc:\n        raise self.retry(exc=exc, countdown=2 ** self.request.retries)\n    return resp.status_code\n",[27,665,666,672,678,690,694,722,726,764,785,792,816,821,835,870],{"__ignoreMap":320},[324,667,668,670],{"class":58,"line":326},[324,669,330],{"class":329},[324,671,334],{"class":333},[324,673,674,676],{"class":58,"line":337},[324,675,330],{"class":329},[324,677,342],{"class":333},[324,679,680,682,685,687],{"class":58,"line":345},[324,681,348],{"class":329},[324,683,684],{"class":333}," celery ",[324,686,330],{"class":329},[324,688,689],{"class":333}," Celery\n",[324,691,692],{"class":58,"line":359},[324,693,376],{"emptyLinePlaceholder":375},[324,695,696,699,701,704,707,709,712,714,717,720],{"class":58,"line":372},[324,697,698],{"class":333},"celery_app ",[324,700,427],{"class":329},[324,702,703],{"class":333}," Celery(",[324,705,706],{"class":436},"\"jobs\"",[324,708,608],{"class":333},[324,710,711],{"class":443},"broker",[324,713,427],{"class":329},[324,715,716],{"class":333},"os.getenv(",[324,718,719],{"class":436},"\"CELERY_BROKER_URL\"",[324,721,614],{"class":333},[324,723,724],{"class":58,"line":379},[324,725,376],{"emptyLinePlaceholder":375},[324,727,728,731,734,737,739,742,744,747,749,752,754,757,759,762],{"class":58,"line":414},[324,729,730],{"class":388},"@celery_app.task",[324,732,733],{"class":333},"(",[324,735,736],{"class":443},"name",[324,738,427],{"class":329},[324,740,741],{"class":436},"\"jobs.deliver_webhook\"",[324,743,608],{"class":333},[324,745,746],{"class":443},"bind",[324,748,427],{"class":329},[324,750,751],{"class":395},"True",[324,753,608],{"class":333},[324,755,756],{"class":443},"max_retries",[324,758,427],{"class":329},[324,760,761],{"class":395},"5",[324,763,460],{"class":333},[324,765,766,768,770,773,775,777,779,781,783],{"class":58,"line":421},[324,767,659],{"class":329},[324,769,389],{"class":388},[324,771,772],{"class":333},"(self, url: ",[324,774,396],{"class":395},[324,776,399],{"class":333},[324,778,402],{"class":395},[324,780,405],{"class":333},[324,782,408],{"class":395},[324,784,411],{"class":333},[324,786,787,790],{"class":58,"line":463},[324,788,789],{"class":329},"    try",[324,791,411],{"class":333},[324,793,794,797,799,802,804,806,808,810,812,814],{"class":58,"line":469},[324,795,796],{"class":333},"        resp ",[324,798,427],{"class":329},[324,800,801],{"class":333}," httpx.post(url, ",[324,803,444],{"class":443},[324,805,427],{"class":329},[324,807,449],{"class":333},[324,809,452],{"class":443},[324,811,427],{"class":329},[324,813,457],{"class":395},[324,815,460],{"class":333},[324,817,818],{"class":58,"line":478},[324,819,820],{"class":333},"        resp.raise_for_status()\n",[324,822,823,826,829,832],{"class":58,"line":483},[324,824,825],{"class":329},"    except",[324,827,828],{"class":333}," httpx.HTTPError ",[324,830,831],{"class":329},"as",[324,833,834],{"class":333}," exc:\n",[324,836,837,840,843,846,849,851,854,857,859,862,865,867],{"class":58,"line":496},[324,838,839],{"class":329},"        raise",[324,841,842],{"class":395}," self",[324,844,845],{"class":333},".retry(",[324,847,848],{"class":443},"exc",[324,850,427],{"class":329},[324,852,853],{"class":333},"exc, ",[324,855,856],{"class":443},"countdown",[324,858,427],{"class":329},[324,860,861],{"class":395},"2",[324,863,864],{"class":329}," **",[324,866,842],{"class":395},[324,868,869],{"class":333},".request.retries)\n",[324,871,872,874],{"class":58,"line":512},[324,873,472],{"class":329},[324,875,475],{"class":333},[14,877,878,879,882],{},"Both work. The arq version reuses one connection-pooled client across every job and never leaves the event loop, so a worker configured with ",[27,880,881],{},"max_jobs = 100"," can have a hundred webhook deliveries in flight concurrently on a single process. The Celery version is a blocking call inside a sync worker: one task occupies one worker slot for the full duration of the request, doing nothing but waiting on the socket. For an async API where tasks are dominated by network I\u002FO, the arq style is not just tidier — it is dramatically more efficient with the same hardware, which is the whole point of the next section.",[32,884,41,889,41,892,41,895,41,898,41,903,41,907,41,913,41,917,41,921,41,928,41,932,41,935,41,939,41,942,41,946,41,949,41,952,41,955,41,958,41,961,41,963,41,966,41,968,41,971,41,974],{"viewBox":885,"role":35,"ariaLabelledBy":886,"xmlns":39,"style":40},"0 0 720 300",[887,888],"cvrva-exec-t","cvrva-exec-d",[43,890,891],{"id":887},"How arq and Celery execute I\u002FO-bound tasks differently",[47,893,894],{"id":888},"Two panels. On the left, an arq worker runs many coroutines concurrently on one event loop, all sharing a single pooled HTTP client while awaiting network I\u002FO. On the right, a Celery prefork worker dedicates one process per task, each blocked waiting on its own socket.",[51,896],{"x":53,"y":53,"width":54,"height":897,"fill":56},"300",[69,899,902],{"x":900,"y":901,"fill":103,"style":104},"180","30","arq: one event loop",[69,904,906],{"x":905,"y":901,"fill":103,"style":104},"540","Celery: one process per task",[58,908],{"x1":909,"y1":910,"x2":909,"y2":911,"stroke":63,"style":912},"360","20","285","stroke-width:1;",[51,914],{"x":61,"y":71,"width":55,"height":915,"rx":916,"fill":97,"stroke":125,"style":64},"215","12",[69,918,920],{"x":900,"y":919,"fill":73,"style":74},"72","shared httpx pool",[51,922],{"x":923,"y":924,"width":925,"height":901,"rx":926,"fill":56,"stroke":125,"style":927},"70","88","220","6","stroke-width:1.5;",[69,929,931],{"x":900,"y":930,"fill":103,"style":84},"108","coroutine 1 · await I\u002FO",[51,933],{"x":923,"y":934,"width":925,"height":901,"rx":926,"fill":56,"stroke":125,"style":927},"126",[69,936,938],{"x":900,"y":937,"fill":103,"style":84},"146","coroutine 2 · await I\u002FO",[51,940],{"x":923,"y":941,"width":925,"height":901,"rx":926,"fill":56,"stroke":125,"style":927},"164",[69,943,945],{"x":900,"y":944,"fill":103,"style":84},"184","coroutine 3 · await I\u002FO",[69,947,948],{"x":900,"y":925,"fill":73,"style":74},"100 in flight, 1 process",[69,950,951],{"x":900,"y":62,"fill":112,"style":84},"idle waits overlap",[51,953],{"x":954,"y":71,"width":55,"height":915,"rx":916,"fill":97,"stroke":98,"style":64},"400",[51,956],{"x":957,"y":924,"width":925,"height":901,"rx":926,"fill":56,"stroke":98,"style":927},"430",[69,959,960],{"x":905,"y":930,"fill":103,"style":84},"worker 1 · blocked",[51,962],{"x":957,"y":934,"width":925,"height":901,"rx":926,"fill":56,"stroke":98,"style":927},[69,964,965],{"x":905,"y":937,"fill":103,"style":84},"worker 2 · blocked",[51,967],{"x":957,"y":941,"width":925,"height":901,"rx":926,"fill":56,"stroke":98,"style":927},[69,969,970],{"x":905,"y":944,"fill":103,"style":84},"worker 3 · blocked",[69,972,973],{"x":905,"y":925,"fill":73,"style":74},"8 processes, 8 in flight",[69,975,977],{"x":905,"y":62,"fill":976,"style":84},"var(--c-yellow)","each waits alone",[134,979,981],{"id":980},"cost-and-performance-at-scale","Cost and performance at scale",[14,983,984],{},"Numbers make the trade-off concrete. Take a common workload: an I\u002FO-bound task that spends roughly 200 ms waiting on an external API — a webhook delivery, an email send, a Stripe call. On a single 2-vCPU box you can run each queue with a sensible default configuration and the ceilings look wildly different.",[14,986,987,988,991,992,995],{},"A Celery prefork worker with 8 processes handles 8 tasks at once. At 200 ms each that is about 40 tasks per second, and the CPUs are almost entirely idle the whole time — you are paying for cores to sit and wait on sockets. RQ forks the same way and lands in the same place. Swap Celery to its ",[27,989,990],{},"gevent"," pool with 100 greenlets and you jump to roughly 500 tasks per second on the same box, because greenlets multiplex the waiting. arq, running coroutines with ",[27,993,994],{},"max_jobs = 300",", comfortably sustains well over 1,000 tasks per second on that single process because the event loop overlaps every idle wait with no per-task process overhead.",[32,997,41,1002,41,1005,41,1008,41,1011,41,1016,41,1019,41,1024,41,1029,41,1035,41,1039,41,1042,41,1045,41,1049,41,1053,41,1058,41,1061,41,1063,41,1067],{"viewBox":998,"role":35,"ariaLabelledBy":999,"xmlns":39,"style":40},"0 0 720 260",[1000,1001],"cvrva-tput-t","cvrva-tput-d",[43,1003,1004],{"id":1000},"Throughput for a 200ms I\u002FO-bound task on one 2-vCPU box",[47,1006,1007],{"id":1001},"Horizontal bar chart of approximate sustained tasks per second: Celery prefork with 8 workers reaches about 40, RQ about 40, Celery with a gevent pool about 500, and arq with 300 concurrent coroutines exceeds 1200.",[51,1009],{"x":53,"y":53,"width":54,"height":1010,"fill":56},"260",[69,1012,1015],{"x":910,"y":1013,"fill":103,"style":1014},"28","font-size:13;font-family:var(--font-sans);","Sustained tasks\u002Fsec · 200ms I\u002FO job · one 2-vCPU box",[58,1017],{"x1":1018,"y1":71,"x2":1018,"y2":925,"stroke":63,"style":927},"200",[69,1020,1023],{"x":1021,"y":923,"fill":103,"style":1022},"196","text-anchor:end;font-size:12;font-family:var(--font-sans);","Celery prefork ×8",[51,1025],{"x":1018,"y":1026,"width":910,"height":1027,"rx":1028,"fill":98},"58","22","3",[69,1030,1034],{"x":1031,"y":1032,"fill":73,"style":1033},"230","74","font-size:11;font-family:var(--font-sans);","~40",[69,1036,1038],{"x":1021,"y":1037,"fill":103,"style":1022},"112","RQ ×8 forks",[51,1040],{"x":1018,"y":1041,"width":910,"height":1027,"rx":1028,"fill":112},"100",[69,1043,1034],{"x":1031,"y":1044,"fill":73,"style":1033},"116",[69,1046,1048],{"x":1021,"y":1047,"fill":103,"style":1022},"154","Celery gevent 100",[51,1050],{"x":1018,"y":1051,"width":1052,"height":1027,"rx":1028,"fill":98},"142","167",[69,1054,1057],{"x":1055,"y":1056,"fill":73,"style":1033},"377","158","~500",[69,1059,1060],{"x":1021,"y":1021,"fill":103,"style":1022},"arq max_jobs 300",[51,1062],{"x":1018,"y":944,"width":954,"height":1027,"rx":1028,"fill":125},[69,1064,1066],{"x":1065,"y":1018,"fill":73,"style":1033},"610","1200+",[69,1068,1070],{"x":1018,"y":62,"fill":73,"style":1069},"font-size:10;font-family:var(--font-sans);","Approximate ceilings for I\u002FO-bound work; CPU-bound tasks invert this entirely.",[14,1072,1073,1074,1077,1078,1082],{},"Translate that to money. If your queue is on a $12\u002Fmonth container and you are pushing a million webhook deliveries a month, arq clears that backlog in a few minutes of the day and the box idles the rest — you never scale past one instance. The same volume on Celery prefork forces you to run 8-to-12 processes across two or three containers just to keep the queue from backing up, tripling the compute line for identical work. The caveat that flips this entirely: if your tasks are ",[165,1075,1076],{},"CPU-bound"," — image resizing, PDF rendering, an ML inference pass — async buys you nothing, because the event loop cannot overlap CPU work. There, Celery's prefork model across real processes is exactly right, and arq's single loop becomes a bottleneck. Match the pool to the workload, not to fashion. For a fuller treatment of pricing math per unit of work, see ",[18,1079,1081],{"href":1080},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fcalculating-cost-per-api-request\u002F","calculating cost per API request",".",[134,1084,1086],{"id":1085},"when-to-choose-each","When to choose each",[14,1088,1089,1092,1093,1095,1096,1100],{},[165,1090,1091],{},"Choose arq when"," your API is FastAPI or another ",[27,1094,29],{}," app and your tasks are I\u002FO-bound — HTTP calls, async DB queries with ",[18,1097,1099],{"href":1098},"\u002Fscaling-and-operating-production-python-apis\u002Fasync-database-access-with-sqlalchemy\u002F","SQLAlchemy",", Redis work. You get async-native tasks, built-in cron scheduling, and a single worker command. It is the lowest-friction choice for the stack most builders on this site run, and it shares your app's connection pools instead of opening a second set.",[14,1102,1103,1106],{},[165,1104,1105],{},"Choose RQ when"," your app is synchronous, your needs are modest, and you want the smallest possible learning curve. RQ is a few hundred lines of obvious code over Redis — easy to read, easy to debug — but it cannot schedule without an add-on and it has no async story.",[14,1108,1109,1112,1113,1117,1118,1082],{},[165,1110,1111],{},"Choose Celery when"," you need features the others lack: multiple broker backends (RabbitMQ, SQS), complex task workflows (chains, groups, chords), fine-grained routing across many queues, or its deep tuning knobs for CPU-bound throughput. It is the heaviest to operate, but nothing else matches its maturity for ",[18,1114,1116],{"href":1115},"\u002Fautomating-side-hustle-operations-with-apis\u002Fprocessing-webhooks-with-python\u002F","processing webhooks"," at volume or running intricate nightly pipelines that feed ",[18,1119,1121],{"href":1120},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdesigning-api-pricing-tiers\u002Fhow-to-charge-for-api-access-using-stripe\u002F","metered billing",[1123,1124,1126],"h3",{"id":1125},"when-to-avoid-each","When to avoid each",[14,1128,1129,1130,1134],{},"Avoid Celery if your whole job system is \"send an email after signup\" — you will spend more time on beat, workers, and Flower than on the feature. Avoid RQ if you are async or need scheduling out of the box. Avoid arq if you need a non-Redis broker or Celery's workflow primitives, or if your tasks are heavy CPU grinders where a single event loop starves. And if all you need is periodic scheduling with no queue at all, weigh a lighter scheduler first — the ",[18,1131,1133],{"href":1132},"\u002Fautomating-side-hustle-operations-with-apis\u002Fscheduling-data-pipelines-with-cron\u002Fapscheduler-vs-celery-beat\u002F","APScheduler vs Celery beat"," comparison covers when an in-process scheduler beats standing up a broker.",[134,1136,1138],{"id":1137},"migration-path","Migration path",[14,1140,1141,1142,1145,1146,1149,1150,1153,1154,1157,1158,1161,1162,1166],{},"Moving between them is mostly mechanical because all three model a task as \"a named function plus serialized arguments.\" From RQ or Celery to arq, you rewrite each ",[27,1143,1144],{},"def task(...)"," as ",[27,1147,1148],{},"async def task(ctx, ...)",", swap synchronous clients for async ones, and replace ",[27,1151,1152],{},".delay()"," \u002F ",[27,1155,1156],{},".enqueue()"," calls with ",[27,1159,1160],{},"await pool.enqueue_job(...)",". The risky part is not the code — it is the cutover. Run both queues side by side: point new enqueues at the new queue while the old workers drain whatever is already in the old one. Keep your idempotency keys identical across the move so a task that happens to run on both systems during the overlap stays safe and never double-charges a customer or double-sends an email. Only after the old queue reads empty and stays empty do you retire its workers. Treat this exactly like a ",[18,1163,1165],{"href":1164},"\u002Fbuilding-monetizing-api-driven-micro-saas\u002Fdeploying-apis-to-render-or-vercel\u002Fzero-downtime-deploys-for-python-apis\u002F","zero-downtime deploy",": the old and new paths coexist until you have proof the new one is healthy.",[134,1168,1170],{"id":1169},"builder-verdict","Builder verdict",[14,1172,1173,1174,1176,1177,1179,1180,1182],{},"For a new async-native FastAPI service moving I\u002FO-bound work off the request path, reach for ",[165,1175,129],{}," first. It shares your event loop and async clients, schedules cron jobs without an add-on, runs from a single command, and — as the throughput numbers show — does far more per dollar of compute for the network-waiting work that dominates most APIs. Step up to ",[165,1178,105],{}," when you genuinely need its muscle: a non-Redis broker like SQS, complex multi-step workflows, or heavy CPU-bound tasks where real processes beat a single loop. ",[165,1181,117],{}," is the comfortable middle only if your app is firmly synchronous and you want the least possible ceremony. Most builders here are async, so the honest default is arq, with Celery held in reserve for the day genuine complexity actually arrives — not the day you imagine it might.",[134,1184,1186],{"id":1185},"faq","FAQ",[14,1188,1189,1192],{},[165,1190,1191],{},"How much does the queue itself cost to run at a million jobs a month?","\nFor I\u002FO-bound work on arq, a single $10-to-$12 container plus the Redis you already pay for handles a million webhook-sized jobs a month with the box idle most of the day. The same volume on Celery prefork typically needs three or more times the compute because each blocked worker wastes a core waiting on sockets, so the honest cost difference is the multiple of containers you run to keep the queue drained.",[14,1194,1195,1198],{},[165,1196,1197],{},"Is arq production-ready, or just a toy?","\nIt is production-ready for async I\u002FO-bound workloads and runs serious traffic, but its ecosystem is far smaller than Celery's. If you hit an edge case you will find fewer answers online and fewer third-party integrations, so budget for reading arq's source rather than a Stack Overflow reply. For the async webhook-and-email workloads most builders start with, that trade is easily worth it.",[14,1200,1201,1204,1205,1208],{},[165,1202,1203],{},"Can I run async code in a Celery task without switching queues?","\nYes, but awkwardly — you wrap the coroutine with ",[27,1206,1207],{},"asyncio.run()"," inside the sync task, which spins up a fresh event loop per task and cannot cleanly reuse a shared async client or connection pool. It works for occasional async calls; it does not give you arq's efficiency, and under load the per-task loop setup adds real overhead.",[14,1210,1211,1214],{},[165,1212,1213],{},"What is the migration risk if I outgrow my first choice?","\nLow, because all three serialize the same \"named function plus arguments\" shape. The only genuine risk is double execution during cutover, which you neutralize by keeping idempotency keys identical and running both queues in parallel until the old one drains. Never flip everything at once; drain, verify empty, then retire.",[14,1216,1217,1220],{},[165,1218,1219],{},"Which queue keeps my compute bill lowest as I scale key-rotation and retries?","\nFor retry-heavy, I\u002FO-bound work arq wins on cost because failed-and-retried jobs still overlap their waits on one loop instead of each pinning a process. If your retries involve CPU-bound processing, Celery's prefork pool scales more predictably across cores — the right answer tracks whether your dominant cost is waiting or computing.",[134,1222,1224],{"id":1223},"related","Related",[14,1226,1227],{},[165,1228,1229],{},"Same track:",[1231,1232,1233,1239,1245,1251],"ul",{},[1234,1235,1236,1238],"li",{},[18,1237,21],{"href":20}," — the parent guide with full worker, beat, and retry setup.",[1234,1240,1241,1244],{},[18,1242,1243],{"href":292},"Caching Python API Responses with Redis"," — the broker all three lean on, doing double duty as your cache.",[1234,1246,1247,1250],{},[18,1248,1249],{"href":1098},"Async Database Access with SQLAlchemy"," — why arq tasks can share your API's async DB pool.",[1234,1252,1253,1257],{},[18,1254,1256],{"href":1255},"\u002Fscaling-and-operating-production-python-apis\u002F","Scaling and Operating Production Python APIs"," — the wider operations area this sits in.",[14,1259,1260],{},[165,1261,1262],{},"Other tracks:",[1231,1264,1265,1270,1276],{},[1234,1266,1267,1269],{},[18,1268,1133],{"href":1132}," — when you need scheduling but not a full queue.",[1234,1271,1272,1275],{},[18,1273,1274],{"href":1115},"Processing Webhooks with Python"," — the classic I\u002FO-bound job these queues were built to run.",[1234,1277,1278,1281],{},[18,1279,1280],{"href":1080},"Calculating Cost per API Request"," — turn the throughput numbers here into margin.",[1283,1284,1285],"style",{},"html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":320,"searchDepth":337,"depth":337,"links":1287},[1288,1289,1290,1291,1294,1295,1296,1297],{"id":136,"depth":337,"text":137},{"id":296,"depth":337,"text":297},{"id":980,"depth":337,"text":981},{"id":1085,"depth":337,"text":1086,"children":1292},[1293],{"id":1125,"depth":345,"text":1126},{"id":1137,"depth":337,"text":1138},{"id":1169,"depth":337,"text":1170},{"id":1185,"depth":337,"text":1186},{"id":1223,"depth":337,"text":1224},"Compare Celery, RQ, and arq for Python background jobs: async support, brokers, scheduling, real throughput numbers, and ops cost. Pick the right queue for your API.","md",{"pageTitle":1301,"type":1302,"datePublished":1303,"dateModified":1304},"Celery vs RQ vs arq: Python Background Job Queues Compared","article","2026-06-18","2026-07-23","\u002Fscaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002Fcelery-vs-rq-vs-arq",{"title":5,"description":1298},"scaling-and-operating-production-python-apis\u002Frunning-background-jobs-with-celery\u002Fcelery-vs-rq-vs-arq\u002Findex","eWOffV51_EVRWstxSEYUmdN2ZmL13GdZzAtTwIvjAd4",{"@context":1310,"@type":1311,"mainEntity":1312},"https:\u002F\u002Fschema.org","FAQPage",[1313,1318,1321,1324,1327],{"@type":1314,"name":1191,"acceptedAnswer":1315},"Question",{"@type":1316,"text":1317},"Answer","For I\u002FO-bound work on arq, a single $10-to-$12 container plus the Redis you already pay for handles a million webhook-sized jobs a month with the box idle most of the day. The same volume on Celery prefork typically needs three or more times the compute because each blocked worker wastes a core waiting on sockets, so the honest cost difference is the multiple of containers you run to keep the queue drained.",{"@type":1314,"name":1197,"acceptedAnswer":1319},{"@type":1316,"text":1320},"It is production-ready for async I\u002FO-bound workloads and runs serious traffic, but its ecosystem is far smaller than Celery's. If you hit an edge case you will find fewer answers online and fewer third-party integrations, so budget for reading arq's source rather than a Stack Overflow reply. For the async webhook-and-email workloads most builders start with, that trade is easily worth it.",{"@type":1314,"name":1203,"acceptedAnswer":1322},{"@type":1316,"text":1323},"Yes, but awkwardly — you wrap the coroutine with asyncio.run() inside the sync task, which spins up a fresh event loop per task and cannot cleanly reuse a shared async client or connection pool. It works for occasional async calls; it does not give you arq's efficiency, and under load the per-task loop setup adds real overhead.",{"@type":1314,"name":1213,"acceptedAnswer":1325},{"@type":1316,"text":1326},"Low, because all three serialize the same \"named function plus arguments\" shape. The only genuine risk is double execution during cutover, which you neutralize by keeping idempotency keys identical and running both queues in parallel until the old one drains. Never flip everything at once; drain, verify empty, then retire.",{"@type":1314,"name":1219,"acceptedAnswer":1328},{"@type":1316,"text":1329},"For retry-heavy, I\u002FO-bound work arq wins on cost because failed-and-retried jobs still overlap their waits on one loop instead of each pinning a process. If your retries involve CPU-bound processing, Celery's prefork pool scales more predictably across cores — the right answer tracks whether your dominant cost is waiting or computing.",1784887028418]