Celery vs RQ vs arq: Choosing a Python Background Job Queue

Once you decide to offload slow work off your request path, the next question is which queue to run it on. The three names that come up are Celery, RQ, and arq — and they sit at very different points on the spectrum from "battle-tested but heavy" to "tiny and async-native." Getting this choice wrong is expensive: you either over-build a distributed task system for a job that fires three times a day, or you bolt a synchronous worker onto an async API and quietly burn threads and money on every request. This comparison helps you pick before you wire anything in. Part of the Running Background Jobs with Celery guide.

The deciding factor for most builders is two things at once: whether your API is async (FastAPI on asyncio) and how much operational machinery you are willing to pay for and keep alive. Everything below flows from those two axes. If you are running an async stack — and most people shipping a first or second commercial API today are — the answer skews hard toward arq, and the rest of this page is about knowing when it does not.

Side-by-side comparison

The headline split is about where your task code runs. Celery and RQ execute synchronous workers, so an async HTTP call inside a task has to run in a thread pool or a per-task event loop you spin up by hand. arq runs your task as a coroutine on the same asyncio machinery FastAPI already uses, which means it shares your async HTTP client, async database drivers, and connection pools with zero adaptation. That single fact — coroutine versus blocking function — cascades into throughput, cost, and how much code you write to keep the two worlds in sync.

Broker flexibility is the other real dividing line. RQ and arq both require Redis and only Redis, which is a feature, not a limitation, for most builders — you almost certainly already run Redis for caching, so the queue costs you nothing extra to stand up. Celery is the only one that speaks RabbitMQ and Amazon SQS. If your architecture already commits to one of those, or you need SQS's managed durability so you never operate a broker yourself, that alone can decide the question before any code is written.

Contrasting an arq task with a Celery task

The same job — POST a payload to a webhook — looks meaningfully different in each. Here is arq, where the task is async def and uses a shared async HTTP client directly. Notice the ctx dict: arq threads a per-worker context through every job, which is where you stash long-lived resources like a pooled httpx client or a database engine.

import os
import httpx
from arq import create_pool
from arq.connections import RedisSettings

async def deliver_webhook(ctx, url: str, payload: dict) -> int:
    # ctx["session"] is a shared httpx.AsyncClient created once on startup
    resp = await ctx["session"].post(url, json=payload, timeout=10.0)
    resp.raise_for_status()
    return resp.status_code

async def startup(ctx):
    ctx["session"] = httpx.AsyncClient()

async def shutdown(ctx):
    await ctx["session"].aclose()

class WorkerSettings:
    functions = [deliver_webhook]
    on_startup = startup
    on_shutdown = shutdown
    max_jobs = int(os.getenv("ARQ_MAX_JOBS", "100"))
    redis_settings = RedisSettings.from_dsn(os.getenv("REDIS_URL"))

# Enqueue from a FastAPI route:
# pool = await create_pool(WorkerSettings.redis_settings)
# await pool.enqueue_job("deliver_webhook", url, payload)

And the Celery equivalent, where the task is a regular def and you make a synchronous call. To get the same resilience you lean on Celery's built-in retry machinery rather than an external library:

import os
import httpx
from celery import Celery

celery_app = Celery("jobs", broker=os.getenv("CELERY_BROKER_URL"))

@celery_app.task(name="jobs.deliver_webhook", bind=True, max_retries=5)
def deliver_webhook(self, url: str, payload: dict) -> int:
    try:
        resp = httpx.post(url, json=payload, timeout=10.0)
        resp.raise_for_status()
    except httpx.HTTPError as exc:
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)
    return resp.status_code

Both work. The arq version reuses one connection-pooled client across every job and never leaves the event loop, so a worker configured with max_jobs = 100 can have a hundred webhook deliveries in flight concurrently on a single process. The Celery version is a blocking call inside a sync worker: one task occupies one worker slot for the full duration of the request, doing nothing but waiting on the socket. For an async API where tasks are dominated by network I/O, the arq style is not just tidier — it is dramatically more efficient with the same hardware, which is the whole point of the next section.

Cost and performance at scale

Numbers make the trade-off concrete. Take a common workload: an I/O-bound task that spends roughly 200 ms waiting on an external API — a webhook delivery, an email send, a Stripe call. On a single 2-vCPU box you can run each queue with a sensible default configuration and the ceilings look wildly different.

A Celery prefork worker with 8 processes handles 8 tasks at once. At 200 ms each that is about 40 tasks per second, and the CPUs are almost entirely idle the whole time — you are paying for cores to sit and wait on sockets. RQ forks the same way and lands in the same place. Swap Celery to its gevent pool with 100 greenlets and you jump to roughly 500 tasks per second on the same box, because greenlets multiplex the waiting. arq, running coroutines with max_jobs = 300, comfortably sustains well over 1,000 tasks per second on that single process because the event loop overlaps every idle wait with no per-task process overhead.

Translate that to money. If your queue is on a $12/month container and you are pushing a million webhook deliveries a month, arq clears that backlog in a few minutes of the day and the box idles the rest — you never scale past one instance. The same volume on Celery prefork forces you to run 8-to-12 processes across two or three containers just to keep the queue from backing up, tripling the compute line for identical work. The caveat that flips this entirely: if your tasks are CPU-bound — image resizing, PDF rendering, an ML inference pass — async buys you nothing, because the event loop cannot overlap CPU work. There, Celery's prefork model across real processes is exactly right, and arq's single loop becomes a bottleneck. Match the pool to the workload, not to fashion. For a fuller treatment of pricing math per unit of work, see calculating cost per API request.

When to choose each

Choose arq when your API is FastAPI or another asyncio app and your tasks are I/O-bound — HTTP calls, async DB queries with SQLAlchemy, Redis work. You get async-native tasks, built-in cron scheduling, and a single worker command. It is the lowest-friction choice for the stack most builders on this site run, and it shares your app's connection pools instead of opening a second set.

Choose RQ when your app is synchronous, your needs are modest, and you want the smallest possible learning curve. RQ is a few hundred lines of obvious code over Redis — easy to read, easy to debug — but it cannot schedule without an add-on and it has no async story.

Choose Celery when you need features the others lack: multiple broker backends (RabbitMQ, SQS), complex task workflows (chains, groups, chords), fine-grained routing across many queues, or its deep tuning knobs for CPU-bound throughput. It is the heaviest to operate, but nothing else matches its maturity for processing webhooks at volume or running intricate nightly pipelines that feed metered billing.

When to avoid each

Avoid Celery if your whole job system is "send an email after signup" — you will spend more time on beat, workers, and Flower than on the feature. Avoid RQ if you are async or need scheduling out of the box. Avoid arq if you need a non-Redis broker or Celery's workflow primitives, or if your tasks are heavy CPU grinders where a single event loop starves. And if all you need is periodic scheduling with no queue at all, weigh a lighter scheduler first — the APScheduler vs Celery beat comparison covers when an in-process scheduler beats standing up a broker.

Migration path

Moving between them is mostly mechanical because all three model a task as "a named function plus serialized arguments." From RQ or Celery to arq, you rewrite each def task(...) as async def task(ctx, ...), swap synchronous clients for async ones, and replace .delay() / .enqueue() calls with await pool.enqueue_job(...). The risky part is not the code — it is the cutover. Run both queues side by side: point new enqueues at the new queue while the old workers drain whatever is already in the old one. Keep your idempotency keys identical across the move so a task that happens to run on both systems during the overlap stays safe and never double-charges a customer or double-sends an email. Only after the old queue reads empty and stays empty do you retire its workers. Treat this exactly like a zero-downtime deploy: the old and new paths coexist until you have proof the new one is healthy.

Builder verdict

For a new async-native FastAPI service moving I/O-bound work off the request path, reach for arq first. It shares your event loop and async clients, schedules cron jobs without an add-on, runs from a single command, and — as the throughput numbers show — does far more per dollar of compute for the network-waiting work that dominates most APIs. Step up to Celery when you genuinely need its muscle: a non-Redis broker like SQS, complex multi-step workflows, or heavy CPU-bound tasks where real processes beat a single loop. RQ is the comfortable middle only if your app is firmly synchronous and you want the least possible ceremony. Most builders here are async, so the honest default is arq, with Celery held in reserve for the day genuine complexity actually arrives — not the day you imagine it might.

FAQ

How much does the queue itself cost to run at a million jobs a month? For I/O-bound work on arq, a single $10-to-$12 container plus the Redis you already pay for handles a million webhook-sized jobs a month with the box idle most of the day. The same volume on Celery prefork typically needs three or more times the compute because each blocked worker wastes a core waiting on sockets, so the honest cost difference is the multiple of containers you run to keep the queue drained.

Is arq production-ready, or just a toy? It is production-ready for async I/O-bound workloads and runs serious traffic, but its ecosystem is far smaller than Celery's. If you hit an edge case you will find fewer answers online and fewer third-party integrations, so budget for reading arq's source rather than a Stack Overflow reply. For the async webhook-and-email workloads most builders start with, that trade is easily worth it.

Can I run async code in a Celery task without switching queues? Yes, but awkwardly — you wrap the coroutine with asyncio.run() inside the sync task, which spins up a fresh event loop per task and cannot cleanly reuse a shared async client or connection pool. It works for occasional async calls; it does not give you arq's efficiency, and under load the per-task loop setup adds real overhead.

What is the migration risk if I outgrow my first choice? Low, because all three serialize the same "named function plus arguments" shape. The only genuine risk is double execution during cutover, which you neutralize by keeping idempotency keys identical and running both queues in parallel until the old one drains. Never flip everything at once; drain, verify empty, then retire.

Which queue keeps my compute bill lowest as I scale key-rotation and retries? For retry-heavy, I/O-bound work arq wins on cost because failed-and-retried jobs still overlap their waits on one loop instead of each pinning a process. If your retries involve CPU-bound processing, Celery's prefork pool scales more predictably across cores — the right answer tracks whether your dominant cost is waiting or computing.

Related

Same track:

• Running Background Jobs with Celery — the parent guide with full worker, beat, and retry setup.
• Caching Python API Responses with Redis — the broker all three lean on, doing double duty as your cache.
• Async Database Access with SQLAlchemy — why arq tasks can share your API's async DB pool.
• Scaling and Operating Production Python APIs — the wider operations area this sits in.

Other tracks:

• APScheduler vs Celery beat — when you need scheduling but not a full queue.
• Processing Webhooks with Python — the classic I/O-bound job these queues were built to run.
• Calculating Cost per API Request — turn the throughput numbers here into margin.