Zero-Downtime Deploys for Python APIs

Every deploy of a paid API is a moment where you can hand your customers a 502 and a reason to ask for a refund. Part of the Deploying APIs to Render or Vercel guide, this page resolves one decision: what you have to build into a Python service so that shipping four times a day costs your users exactly zero failed requests. The answer is four mechanisms working together — a readiness gate the platform can trust, graceful shutdown that drains in-flight work, schema migrations ordered so old and new code both run against the same database, and a rollback trigger that fires without a human in the loop.

None of this is platform magic. Render, Railway, and Fly all run rolling deploys by default, and all of them will happily roll a broken build straight into production if your app lies about being healthy. The platform's job is to shift traffic; your job is to tell it the truth about when to shift it.

What actually breaks during a deploy

A naive deploy is a stop-then-start: the old process gets killed, the new one boots, and every request that lands in between hits a closed socket. On a Python API that boot window is not trivial. Importing FastAPI, Pydantic models, and a SQLAlchemy metadata tree costs a second or two on a small instance; opening the database pool and warming the first connection costs more. Eight seconds of hard downtime per deploy, four deploys a day, is roughly thirty-two seconds of outage daily — enough to blow a 99.9% uptime commitment inside a month and enough to make webhook senders start retrying.

A rolling deploy fixes the arithmetic by overlapping: the new instance boots and proves itself healthy while the old one is still serving, and only then does the old one start refusing new work. The overlap window is where all the engineering lives. Get it wrong in either direction and you either cut requests mid-flight or leave two incompatible versions of your code fighting over the same tables.

Stop-and-start deploy versus rolling deploy timeline A stop-then-start release leaves an eight second gap where requests fail, while a rolling release overlaps the old and new instances so every request is served. Request availability across one release Stop, then start v1 serving 502 / 503 v2 serving about 8 s of dropped traffic Rolling with drain v1 serving, then draining v2 ready, then serving overlap window zero failed requests

Health checks and readiness gates

Ship two endpoints, not one. Liveness answers "is this process alive enough to keep?" and must never touch the database — if it does, a brief Postgres hiccup makes the platform kill and restart every healthy instance you own at exactly the moment the database is struggling. Readiness answers "should the load balancer send me traffic right now?" and is allowed to depend on the pool, the cache, and a boot-complete flag.

The readiness gate is what turns a rolling deploy from hopeful into safe. The new instance returns 503 from /readyz until its async SQLAlchemy engine has opened and verified a connection. The platform holds traffic back for exactly that long, then promotes it. Crucially the same flag flips back to false the moment shutdown starts, which is how you tell the router to stop sending you work before you stop being able to do it.

Python
# app/health.py
import asyncio
import os
from contextlib import asynccontextmanager

from fastapi import APIRouter, FastAPI, Response
from sqlalchemy import text
from sqlalchemy.ext.asyncio import create_async_engine

DATABASE_URL = os.environ["DATABASE_URL"]
DRAIN_SECONDS = float(os.getenv("DRAIN_SECONDS", "5"))
RELEASE_SHA = os.getenv("RELEASE_SHA", "dev")

engine = create_async_engine(DATABASE_URL, pool_pre_ping=True)
router = APIRouter()


class Gate:
    """Single source of truth for 'should I receive traffic?'."""
    ready = False


@asynccontextmanager
async def lifespan(app: FastAPI):
    async with engine.connect() as conn:
        await conn.execute(text("SELECT 1"))
    Gate.ready = True
    yield
    # Shutdown begins: fail readiness first, keep serving in-flight work.
    Gate.ready = False
    await asyncio.sleep(DRAIN_SECONDS)
    await engine.dispose()


@router.get("/healthz", include_in_schema=False)
async def healthz() -> dict[str, str]:
    """Liveness: process-only, never touches I/O."""
    return {"status": "alive", "release": RELEASE_SHA}


@router.get("/readyz", include_in_schema=False)
async def readyz(response: Response) -> dict[str, str]:
    """Readiness: the routing decision."""
    if not Gate.ready:
        response.status_code = 503
        return {"status": "draining", "release": RELEASE_SHA}
    return {"status": "ready", "release": RELEASE_SHA}


app = FastAPI(lifespan=lifespan)
app.include_router(router)

Point the platform's health check path at /readyz, not /healthz — Render's healthCheckPath, Fly's [[http_service.checks]], and Railway's healthcheck field all gate traffic on it. Returning the release SHA is worth the two lines: curl the public URL in a loop during a deploy and watch the ratio of old to new SHAs shift, the cheapest possible proof that the overlap works.

Instance lifecycle as the readiness probe sees it An instance moves from boot to warm-up to ready to draining to exited, and the load balancer routes traffic only while the readiness probe returns 200. What /readyz returns at each stage probe passes SIGTERM Boot no socket Warm-up 503 Ready 200 Draining 503 Exited gone new traffic routed only here in-flight requests still finish while draining

Graceful shutdown: draining in-flight requests

Flipping the readiness flag is only half of a clean exit. The platform sends SIGTERM, waits a grace period, then sends SIGKILL. Uvicorn handles SIGTERM by refusing new connections and waiting for open ones to complete, but it only waits as long as you let it — and the default grace period on most platforms is short enough to cut a slow report endpoint in half.

Three numbers have to be ordered correctly, and almost every broken drain I have debugged comes down to getting them backwards. Your longest realistic request must be shorter than Uvicorn's graceful-shutdown timeout, which must be shorter than the platform's kill grace period. Add the readiness-flip delay on top, because the router needs a probe cycle or two to notice you went unhealthy before it stops sending new work.

Dockerfile
# Dockerfile — the drain contract, expressed as flags
ENV DRAIN_SECONDS=5
CMD ["sh", "-c", "exec uvicorn app.health:app \
  --host 0.0.0.0 --port ${PORT:-8000} \
  --workers ${WEB_CONCURRENCY:-2} \
  --timeout-graceful-shutdown ${GRACEFUL_TIMEOUT:-25} \
  --timeout-keep-alive ${KEEPALIVE:-5}"]

The exec matters more than it looks: without it the shell becomes PID 1, swallows SIGTERM, and your carefully written lifespan shutdown never runs. Set the platform's own grace to something comfortably larger — 30 to 40 seconds against a 25-second Uvicorn timeout — so the process always exits on its own terms rather than being killed mid-response. If you run multiple workers, the same reasoning applies per worker; the worker-count decision itself is covered in Uvicorn vs Gunicorn worker configuration.

Long-running work is the trap. If an endpoint holds a connection for ninety seconds rendering a PDF, no grace period saves you. Push that work to a queue and return a job id — background jobs with Celery exist precisely so your HTTP layer always drains in under thirty seconds. Webhook consumers need the same discipline; building an idempotent webhook receiver is what makes a mid-deploy retry harmless.

Graceful shutdown sequence after SIGTERM The platform sends SIGTERM, the API process fails its readiness probe, the load balancer stops routing new requests, in-flight work finishes, and the process exits cleanly. Platform API process Load balancer SIGTERM Gate.ready = False GET /readyz gives 503 stop new routing drain in-flight, max 25 s exit 0, before SIGKILL

Migration ordering: expand, then contract

Rolling deploys guarantee that old code and new code run at the same time against one database, usually for thirty to ninety seconds. Any migration that is not backwards compatible turns that overlap into an outage: rename a column and every request still handled by the old instance raises UndefinedColumn. The rule is that a single deploy may never contain both a schema change and the code that requires it.

Split every breaking change into three releases. Expand adds the new structure while leaving the old one intact — nullable columns, new tables, new indexes built concurrently. Migrate ships code that writes both shapes and reads the new one, plus a backfill for existing rows. Contract removes the old column, days later, once no running instance references it. Each step is individually reversible, which is the property that makes an automatic rollback safe.

StepSchema changeCode that runsReversible
1. ExpandAdd nullable columnv1 and v2 both fineYes, drop it
2. MigrateBackfill rowsv2 dual-writes, reads newYes, v1 still works
3. ContractDrop old columnv2 onlyNo, plan it
Python
"""expand: add plan_code beside plan_id

Revision ID: 20260723_expand_plan_code
"""
import sqlalchemy as sa
from alembic import op

revision = "20260723_expand_plan_code"
down_revision = "20260702_usage_events"


def upgrade() -> None:
    # Nullable: existing rows stay valid, v1 code keeps inserting without it.
    op.add_column("subscriptions", sa.Column("plan_code", sa.Text(), nullable=True))

    # CONCURRENTLY cannot run inside a transaction; Alembic gives us an escape hatch.
    with op.get_context().autocommit_block():
        op.create_index(
            "ix_subscriptions_plan_code",
            "subscriptions",
            ["plan_code"],
            unique=False,
            postgresql_concurrently=True,
        )


def downgrade() -> None:
    with op.get_context().autocommit_block():
        op.drop_index("ix_subscriptions_plan_code", postgresql_concurrently=True)
    op.drop_column("subscriptions", "plan_code")

Two Postgres details decide whether this is invisible or catastrophic. Building an index without CONCURRENTLY takes a lock that blocks writes for the whole build — minutes on a table with millions of usage rows. And any ALTER TABLE waiting behind a long transaction queues every later query behind it too, so set a short lock_timeout on the migration connection specifically and retry rather than letting one statement freeze the API. If releases already cause pool pressure, fixing connection pool exhaustion is the prerequisite read. The same expand-contract discipline applies to your public response bodies, which is the subject of deprecating an endpoint without breaking customers.

Expand, migrate, and contract release ordering Three separate releases: expand adds a nullable column, migrate backfills and switches reads, and contract drops the old column once no old code remains. Release Schema state Code versions it serves 1. Expand Monday old column kept, new nullable column added v1 and v2 both work 2. Migrate Monday rows backfilled, v2 writes both columns v2 reads new, v1 safe 3. Contract Thursday old column dropped, index cleaned up v2 only, no rollback Steps 1 and 2 are reversible; wait days before step 3.

Rollback triggers that fire without you

A deploy is not finished when the platform says "live". It is finished when the new release has survived a watch window. Define that window in code and let a script decide, because at 11pm on a Friday your judgement is the least reliable component in the system.

Three signals are enough for a small API: the 5xx rate over a rolling minute, p95 latency against the pre-deploy baseline, and readiness flapping — an instance oscillating between 200 and 503 is a boot loop dressed up as a rolling deploy. Any one tripping should roll back automatically and page you afterwards. Structured logging with structlog is the fastest route to per-release error rates.

Python
# scripts/watch_release.py — run right after the deploy finishes
import asyncio
import os
import sys

import httpx

BASE_URL = os.environ["DEPLOY_BASE_URL"]
RELEASE_SHA = os.environ["RELEASE_SHA"]
WINDOW_SECONDS = int(os.getenv("WATCH_WINDOW_SECONDS", "600"))
MAX_ERROR_RATE = float(os.getenv("MAX_ERROR_RATE", "0.01"))
P95_BUDGET_MS = float(os.getenv("P95_BUDGET_MS", "800"))


async def sample(client: httpx.AsyncClient) -> tuple[float, float, str]:
    """Read the app's own metrics endpoint for the current release."""
    resp = await client.get("/internal/metrics", params={"release": RELEASE_SHA})
    resp.raise_for_status()
    body = resp.json()
    return float(body["error_rate"]), float(body["p95_ms"]), str(body["release"])


async def verdict() -> str:
    deadline = asyncio.get_running_loop().time() + WINDOW_SECONDS
    async with httpx.AsyncClient(base_url=BASE_URL, timeout=10.0) as client:
        while asyncio.get_running_loop().time() < deadline:
            error_rate, p95_ms, live_sha = await sample(client)
            if live_sha != RELEASE_SHA:
                return "flapping"
            if error_rate > MAX_ERROR_RATE:
                return "errors"
            if p95_ms > P95_BUDGET_MS:
                return "latency"
            await asyncio.sleep(15)
    return "healthy"


async def main() -> int:
    match await verdict():
        case "healthy":
            print(f"release {RELEASE_SHA} promoted")
            return 0
        case reason:
            print(f"rolling back {RELEASE_SHA}: {reason}", file=sys.stderr)
            return 1


if __name__ == "__main__":
    raise SystemExit(asyncio.run(main()))

Wire a non-zero exit to your platform's rollback command — render deploys rollback, flyctl releases rollback, or simply redeploying the previous image tag. Immutable image tags are what make this instant: rolling back should re-point traffic at a container that already exists, never rebuild from source. That is one of the strongest arguments for containerizing your Python API with Docker even on a platform that would happily build from a buildpack.

Automatic rollback decision tree During a ten minute watch window, an elevated error rate, doubled p95 latency, or flapping readiness each trigger an automatic rollback to the previous release. Deploy v2, watch 10 min 5xx rate over 1%? yes no p95 over budget? yes no readiness flapping? yes no All clear, promote Roll back to v1 re-point at the previous image tag, under 60 s

When this is worth building, and when it is not

Build the full stack — readiness gate, drain, expand-contract, automatic rollback — the day you take money for the API. The cost is one afternoon and roughly zero ongoing compute; the alternative is refunding customers for failures you caused on purpose. If you are on a paid instance already, running two instances during a rolling deploy costs nothing extra on Render or Railway because the overlap lasts under a minute.

Skip the automatic rollback watcher, but nothing else, when you are pre-revenue on a single instance. A rolling deploy cannot overlap at all unless the platform briefly runs two instances, so check that setting first: on Fly it is min_machines_running plus a rolling strategy, on Render it is any paid plan. Skip the whole thing only for an internal tool where thirty seconds of downtime costs nothing.

The one case that demands more is a high-traffic API where a bad release costs thousands per minute. There, graduate to canary: route 5% of traffic to the new release, compare error rates against the control, and promote or discard. The mechanisms above are its prerequisites — canary routing without a readiness gate and reversible migrations is just a faster way to break production.

The transition path from where you are now

If you are deploying by pushing to main and hoping, four changes get you to zero-downtime in order. First, add /healthz and /readyz and point the platform's health check at readiness. Second, add exec plus --timeout-graceful-shutdown to your start command and raise the platform grace period above it. Third, move migrations out of the app start command into a separate release step, and split the next breaking schema change into expand and contract. Fourth, add the watch script to the end of your deploy job. Each step ships independently. Verify the drain step by running a load generator against staging while you deploy and confirming zero non-2xx responses — an assertion worth keeping once you have tested the endpoints with pytest.

Builder verdict

Rolling deploys with a readiness gate and a graceful drain win for every commercial Python API under roughly a thousand requests per second, and it is not close. You get the availability of blue-green at a fraction of the cost, because the overlap lasts seconds instead of requiring a duplicate environment you pay for continuously. Start with the four mechanisms here, keep migrations expand-contract forever, and buy more sophistication only when a minute of degraded service costs more than the engineering time to prevent it. The highest-leverage code on this page is the readiness flag flip in your shutdown handler — three lines that eliminate most deploy-time 502s on their own.

FAQ

Does running two instances during a rolling deploy double my hosting bill? No. The overlap lasts thirty to ninety seconds per release, so even at ten deploys a day you are paying for a few extra instance-minutes — cents per month on usage-billed platforms, and nothing at all on flat per-service plans where the overlap is included. Compare that with the refund and churn cost of one visibly broken release.

Should I run database migrations as part of the app's start command? Never. Every instance would race to run them simultaneously, and a failed migration would crash-loop your whole service instead of failing one job. Run migrations as a separate release step that completes before the new instances boot, and keep them backwards compatible so the old code surviving the overlap is unaffected.

How long should the drain window be for a metered API? Long enough for your slowest endpoint plus a probe cycle, which for most APIs means twenty to thirty seconds. Check your p99 latency and any billing writes that must complete — a usage event that never lands is revenue you cannot invoice, so the drain has to outlast your longest write path, not your median request.

What is the real risk of skipping the contract step and leaving old columns? Low technical risk, mounting cost. Dead columns bloat rows, slow sequential scans, and confuse the next person reading the model, and unused indexes still get written on every insert. Schedule contract migrations a few days after the expand release so rollback stays possible, then actually run them rather than leaving debt to accumulate.

Can I do zero-downtime deploys while also rotating secrets? Yes, and the pattern is identical: expand before you contract. Add the new credential, deploy code that accepts both old and new, then remove the old one in a later release. The same overlap logic applied to customer credentials is covered in detail in the guide on rotating API keys without downtime.

Same track:

Other tracks: