Monitoring and Logging Python APIs in Production

An API that runs without instrumentation is an API you cannot operate. When a customer reports "it's slow" or your Stripe bill spikes overnight, you need a request_id to grep, a latency histogram to query, and a cost-per-request number to defend your margins — not a guess and a shrug. This guide wires structured logging, Prometheus metrics, and the four numbers that actually matter into a FastAPI service, then shows you how to turn those numbers into alerts before a customer turns them into a churn event. Part of the Scaling and Operating Production Python APIs guide.

The goal here is operational leverage: one consistent log schema you can query, a /metrics endpoint your dashboard scrapes, and enough cost visibility that you can tie a slow endpoint back to an eroded gross margin on your pricing tiers. Observability is not a nice-to-have you bolt on after launch; for a one- or two-person team it is the difference between debugging in minutes and debugging by customer email. You do not have a support rotation or an SRE team to absorb the cost of flying blind, so the instrumentation has to do that work for you.

The three signals: logs, metrics, and traces

Before writing any code, get the mental model right, because most builders overspend on one signal and ignore the other two. Observability rests on three distinct signals, and they answer different questions. Logs are discrete events with full context — they answer "what exactly happened to this one request." Metrics are cheap numeric aggregates sampled over time — they answer "what is happening across all requests right now." Traces stitch a single request's path across services — they answer "which span inside this slow request was actually slow." Reaching for the wrong one wastes money: paging through logs to estimate a p95 is slow and imprecise, and firing a metric per user id will bankrupt your Prometheus server.

The reason to run all three is that each one is nearly useless alone. Metrics show you a spike at 14:32 but cannot tell you which customer or payload caused it. Logs let you reconstruct that one request in detail but cannot show you the trend that told you to look. Traces pinpoint the slow database call but only once metrics have told you a route is slow and logs have handed you the offending request_id. The craft is matching the signal to the question and keeping each one's cost proportional to its value.

For most APIs the right order of investment is logs first, metrics second, traces last. Logs cost nothing to start and pay off on day one when a webhook fails. Metrics are next because they unlock alerting and capacity planning. Traces come last because they earn their keep only once your request fans out across a database, a cache, and one or more upstream APIs — before that, a good log line already tells you where the time went.

Prerequisites

This guide assumes a working FastAPI service on Python 3.11+ and that you are comfortable with making HTTP requests and async route handlers. Install the two instrumentation libraries:

pip install structlog prometheus-client
# optional, for the tracing step:
pip install opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-fastapi

We use structlog for log emission, but every snippet works with the standard library logging module configured with a JSON formatter — the dedicated structlog guide covers the trade-off. Configuration is driven entirely by environment variables, never hardcoded, so the same image runs identically in local development, staging, and production with only the environment changing:

If you deploy in a container, none of this changes — you log to stdout and let the runtime capture the stream, which is exactly the pattern covered in containerizing Python APIs with Docker.

Step 1: JSON structured logging with a request_id

Plain-text logs are unqueryable. The moment you have two concurrent requests, interleaved print() output becomes useless for tracing a single call — line 3 belongs to request A, line 4 to request B, and you have no way to tell them apart. The fix is one JSON object per line, every line carrying a request_id that ties together the entry, exit, and any errors of a single request. Once every line is structured JSON, your log backend can filter by request_id, status, or path the way a database filters rows, and the interleaving stops mattering.

Bind the request_id into a contextvars-backed context so every log call inside the request automatically inherits it, with no parameter threading:

# logging_setup.py
import logging
import os
import structlog

def configure_logging() -> None:
    level = os.getenv("LOG_LEVEL", "INFO").upper()
    logging.basicConfig(format="%(message)s", level=level)
    structlog.configure(
        processors=[
            structlog.contextvars.merge_contextvars,
            structlog.processors.add_log_level,
            structlog.processors.TimeStamper(fmt="iso"),
            structlog.processors.JSONRenderer(),
        ],
        wrapper_class=structlog.make_filtering_bound_logger(
            logging.getLevelName(level)
        ),
        cache_logger_on_first_use=True,
    )

log = structlog.get_logger()

Now add middleware that generates a request_id, binds it, and logs request completion with its status and duration:

# middleware.py
import os
import time
import uuid
import structlog
from fastapi import Request

log = structlog.get_logger()
SERVICE_NAME = os.getenv("SERVICE_NAME", "builder-api")

async def logging_middleware(request: Request, call_next):
    request_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
    structlog.contextvars.bind_contextvars(
        request_id=request_id,
        service=SERVICE_NAME,
        path=request.url.path,
        method=request.method,
    )
    start = time.perf_counter()
    try:
        response = await call_next(request)
    except Exception:
        log.exception("request_failed")
        structlog.contextvars.clear_contextvars()
        raise
    duration_ms = (time.perf_counter() - start) * 1000
    log.info("request_completed", status=response.status_code,
             duration_ms=round(duration_ms, 2))
    response.headers["X-Request-ID"] = request_id
    structlog.contextvars.clear_contextvars()
    return response

Two details in that middleware are load-bearing. First, it honours an inbound X-Request-ID header before generating its own, so when a request arrives from an upstream gateway or a webhook sender that already assigned an id, the trail stays continuous across service boundaries. Second, it clears the context on both the success and the exception path — skip the clear_contextvars() in the error branch and a leaked request_id will bleed into the next request that happens to reuse the same worker, poisoning your logs with the wrong id.

Returning the X-Request-ID header lets a customer quote the exact ID from a failed call, turning a vague bug report into a one-line grep. One caveat with contextvars and async: if you spawn a background coroutine with asyncio.create_task inside a request, it captures the context at creation time, so the request_id follows it — but a task you fire and forget after the response has returned will log against a context you have already cleared, so bind a fresh id for genuinely detached work. For the full processor pipeline and why contextvars beats passing loggers around, see structured logging with structlog.

Step 2: A Prometheus /metrics endpoint with a latency histogram

Logs answer "what happened to this request"; metrics answer "what is happening across all requests". A histogram of request latency is the single most valuable metric you can expose — it powers p95/p99, alerting, and capacity planning, all from one instrument that costs a handful of counter increments per request.

# metrics.py
import os
import time
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
from fastapi import Request, Response

REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds",
    "Request latency in seconds",
    labelnames=("method", "route", "status"),
    buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)
REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total requests",
    labelnames=("method", "route", "status"),
)

async def metrics_middleware(request: Request, call_next):
    start = time.perf_counter()
    response = await call_next(request)
    # Use the route template, NOT request.url.path, to avoid high cardinality.
    route = request.scope.get("route")
    route_label = route.path if route else "unmatched"
    labels = (request.method, route_label, str(response.status_code))
    REQUEST_LATENCY.labels(*labels).observe(time.perf_counter() - start)
    REQUEST_COUNT.labels(*labels).inc()
    return response

async def metrics_endpoint(_: Request) -> Response:
    if os.getenv("METRICS_ENABLED", "true").lower() != "true":
        return Response(status_code=404)
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)

The bucket boundaries are a design decision, not a default to accept blindly. Prometheus histograms compute quantiles by interpolating within the bucket the quantile falls into, so a bucket that is too wide makes your p95 imprecise. The buckets above are tuned for a typical JSON API where most responses land between 25 and 250 milliseconds; if your API is a thin proxy that returns in single-digit milliseconds, add finer buckets down low, and if it does heavy work, extend the top end. Pick boundaries that bracket your real target latency so the percentile lands on a tight bucket rather than a wide one.

Wire everything into the app. Order matters: register the metrics middleware before the logging middleware so the timer brackets the whole stack.

# main.py
import os
from fastapi import FastAPI
from logging_setup import configure_logging
from middleware import logging_middleware
from metrics import metrics_middleware, metrics_endpoint

configure_logging()
app = FastAPI(title=os.getenv("SERVICE_NAME", "builder-api"))
app.middleware("http")(logging_middleware)
app.middleware("http")(metrics_middleware)
app.add_route("/metrics", metrics_endpoint, include_in_schema=False)

@app.get("/work")
async def work():
    return {"ok": True}

The route.path template (/items/{id}) keeps the label count bounded. Using the raw path (/items/42, /items/43, ...) would create one time series per id and melt your Prometheus server — the most common self-inflicted observability outage. A single high-cardinality label can turn a few dozen series into millions in an afternoon, and because Prometheus holds active series in memory, the failure mode is an out-of-memory kill of the whole monitoring stack, not a graceful slowdown. Treat every label value as something you would be comfortable seeing in a dropdown filter.

Step 3: Track p95/p99, error rate, and cost-per-request

The histogram from Step 2 already lets your monitoring backend compute percentiles with a PromQL query — no extra code:

# p95 latency per route over 5 minutes
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route))

# error rate: share of 5xx responses
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

Report p95 and p99, never the average, and this is where most builders go wrong. An average latency hides your worst experiences: if 99 requests return in 40 ms and one takes 4 seconds, the mean is a comfortable-looking 80 ms while a real customer just waited four seconds and may have timed out. The p99 tells you the truth about your tail — the slice of traffic that generates support tickets and churn — and the gap between p50 and p99 tells you how consistent your service is. A p50 of 40 ms with a p99 of 3 s is a service with a serious tail problem hiding behind a healthy median.

The chart above makes the point concrete: the bulk of the 1,132 requests finish under 100 ms, but the p95 sits in the 250–500 ms bucket and the p99 spills into the 500 ms–1 s range. The average would round to roughly 90 ms and quietly ignore the tens of requests that took ten times longer. Alert on the p99, not the mean.

Cost-per-request is the metric that turns observability into a margin defense, and it has to be recorded explicitly because only you know your upstream fees. Track the spend each request incurs — model tokens, third-party API calls, compute — as a counter:

# cost.py
from prometheus_client import Counter

REQUEST_COST_USD = Counter(
    "request_cost_usd_total",
    "Cumulative request cost in USD",
    labelnames=("route", "tier"),
)

def record_cost(route: str, tier: str, *, upstream_calls: int,
                tokens: int) -> None:
    # Wire your real unit economics in here.
    upstream_fee = upstream_calls * 0.002      # $0.002 per upstream call
    token_fee = tokens / 1000 * 0.0006         # $0.60 per 1M tokens
    REQUEST_COST_USD.labels(route=route, tier=tier).inc(
        upstream_fee + token_fee
    )

Dividing cumulative cost by request count per tier gives average cost-per-request. When that number on your free tier creeps toward your paid tier's price, you have a margin problem before the invoice arrives — the same logic behind usage-based pricing. The same counter is the raw signal you would persist to a durable store for customer-facing reporting, which is exactly the pattern in logging API usage events to Postgres. Watch this number alongside your outbound rate-limiting strategy, since retries against an upstream multiply both latency and cost. If your upstream fee is dominated by LLM tokens, the same instrument feeds directly into controlling LLM API costs in production.

Step 4: Turn metrics into alerts and SLOs

A dashboard nobody is watching at 3 a.m. is not monitoring — it is decoration. The point of the histogram and counters is to page you before a customer does. Define one or two service level objectives, express them as PromQL, and let your alerting backend fire when they break. For a small API, two alerts cover most of the ground: a latency SLO on the tail, and an error-budget-style alert on the 5xx rate.

# ALERT: p99 latency over 1s for 10 minutes on any route
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route)) > 1

# ALERT: 5xx error rate above 2% over 5 minutes
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m])) > 0.02

Keep the alert set small on purpose. An inbox that fires ten warnings a day trains you to ignore all of them, and the one that mattered arrives buried. Two or three high-signal alerts that only fire when a real customer is affected beat a wall of noisy thresholds. Tie each alert to a symptom a customer would feel — slow responses, failed requests, a stalled webhook backlog — not to an internal number like CPU that may be high for perfectly healthy reasons. When an alert fires, the request_id in your logs and the route label on the metric together point you straight at the failing calls, and if the slow span is a database round trip, that is your cue to look at async database access with SQLAlchemy and connection pool sizing.

Step 5 (optional): OpenTelemetry tracing

Metrics tell you a route is slow; a trace tells you which span — the database call, the upstream API, or your own code. When a single request fans out to several services, add OpenTelemetry. It is opt-in and gated on the collector endpoint being set:

# tracing.py
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

def configure_tracing(app) -> None:
    endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    if not endpoint:
        return  # tracing stays off until an endpoint is configured
    resource = Resource.create(
        {"service.name": os.getenv("SERVICE_NAME", "builder-api")}
    )
    provider = TracerProvider(resource=resource)
    provider.add_span_processor(
        BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint))
    )
    trace.set_tracer_provider(provider)
    FastAPIInstrumentor.instrument_app(app)

The BatchSpanProcessor exports spans asynchronously off the request path, so it adds negligible latency. Sample aggressively in production — exporting every span at high traffic is both expensive and rarely necessary. A 5–10% head sample keeps a representative view of latency while cutting export volume and your tracing backend's bill by an order of magnitude. The one time you want higher sampling is right after a deploy or during an incident, when you can temporarily raise the rate to catch the specific failure you are chasing, then drop it back.

Configuration reference

Gotchas and failure modes

1. Logging PII. A log.info("login", email=user.email) line ships personal data to a log store you may not control, creating a compliance liability under GDPR or CCPA that can outlast the bug you were debugging. Log a hashed user id, never raw emails, tokens, or card data. Add a structlog processor that redacts known-sensitive keys before the renderer runs, so the redaction is structural rather than something every caller has to remember.
2. High-cardinality metric labels. Putting user ids, request paths with ids, or raw query strings in label values creates unbounded time series and can crash Prometheus. Labels must be low-cardinality: method, route template, status, tier — nothing per-user. If you need per-user cost accounting, that belongs in a log line or a database row, not a metric label.
3. Blocking log I/O. Writing logs synchronously to a slow sink (a remote HTTP endpoint, an unbuffered file) stalls the event loop and tanks tail latency for every concurrent request sharing that worker. Log to stdout and let the platform ship them; never make a network call inside the request path to emit a log.
4. Unsampled traces and debug logs in prod. DEBUG logging at scale buries signal and inflates your log bill; 100% trace sampling does the same to your tracing backend. Default to INFO and single-digit-percent sampling.
5. Counting on request.url.path for metrics. As covered in Step 2, this is the fast path to a cardinality explosion. Always label with the route template.
6. Alerting on averages. A mean latency alert stays green while your p99 quietly triples, because the tail is a rounding error in the average. Alert on the percentile you actually promise customers, not the mean.

Verification

Start the service and confirm both signals. The metrics endpoint should return Prometheus text:

curl -s http://localhost:8000/metrics | grep http_request_duration_seconds
# http_request_duration_seconds_bucket{le="0.1",method="GET",route="/work",status="200"} 1.0
# http_request_duration_seconds_count{method="GET",route="/work",status="200"} 1.0

And a request should emit a single-line JSON log carrying the request_id:

{"request_id": "8f1c...", "service": "builder-api", "path": "/work", "method": "GET", "status": 200, "duration_ms": 3.41, "level": "info", "timestamp": "2026-06-18T10:00:00Z", "event": "request_completed"}

If the log is multi-line plain text, your formatter is not wired up; if /metrics is empty, the middleware is registered after the route that handles /metrics. Fold both checks into your test suite so a regression in the middleware order fails CI rather than production — a request through the app should assert a 200 on /metrics and a parseable JSON log line, which fits naturally into testing Python APIs with pytest.

Cost and performance note

Instrumentation is not free, but the overhead is small and bounded against the alternative of operating blind. A Prometheus histogram observation and an in-process counter increment are microsecond-scale; the JSON log line costs a serialization plus a stdout write. In practice the per-request overhead of logging plus metrics is well under a millisecond — invisible next to a single database round-trip that costs several.

The real cost lever is volume, not per-call work. Put concrete numbers on it: at a steady 50 requests per second you serve roughly 130 million requests a month, and at one INFO log line of a few hundred bytes each that is on the order of 30–60 GB of logs. Most hosted log backends charge a few dollars per GB ingested, so INFO logging costs you a manageable double-digit sum, while flipping to DEBUG — which can emit five to ten lines per request — turns that into hundreds of GB and a bill that dwarfs your compute. Metrics scale with series count rather than request volume, so a handful of low-cardinality labels keeps Prometheus in the noise; the moment a stray high-cardinality label creates millions of series, storage and memory costs jump non-linearly. Keep LOG_LEVEL at INFO, sample traces, and hold your labels low-cardinality, and you get full operational visibility for a rounding error on latency and a predictable, small spend on storage. The blind spot — not knowing your p99 or your cost-per-request until a customer or an invoice tells you — is far more expensive than any of it.

FAQ

How much does monitoring add to my hosting bill at 1M requests a month? At a million requests a month with one INFO log line each, expect on the order of half a gigabyte of logs — a few dollars on most hosted backends — plus effectively nothing for metrics, since Prometheus cost scales with series count, not request volume. The overhead only balloons if you leave DEBUG logging on or let a high-cardinality label multiply your metric series; both are configuration mistakes, not inherent costs, and both are avoidable with the defaults in this guide.

Do I need both structlog and Prometheus, or is one enough? They answer different questions and you want both. Structured logs give you per-request detail you can grep by request_id for debugging; Prometheus metrics give you aggregate trends, percentiles, and alerting across all traffic. Logs without metrics mean no dashboards or alerts; metrics without logs mean you can see a spike but cannot trace a single failing request. Together they cost a few dollars a month at small scale, which is far cheaper than one prolonged outage you debug by guesswork.

How do I compute p95 and p99 without writing percentile code? Expose a Prometheus histogram (Step 2) and use histogram_quantile(0.95, ...) in PromQL. The histogram buckets accumulate in-process; your monitoring backend does the percentile math at query time, so you never compute or store percentiles yourself. The one thing you control is the bucket boundaries — set them to bracket your real target latency so the quantile lands on a tight bucket rather than a wide one, or the number will be imprecise.

Should the /metrics endpoint be public? No. It leaks operational detail and route names that help an attacker map your service. Keep it reachable only from your scraper — bind it to an internal network, restrict it at the reverse proxy by IP, or require an auth header. The METRICS_ENABLED flag lets you disable it entirely in environments that should not expose it at all.

Does cost-per-request tracking need a separate billing system? Not to observe it. The Prometheus counter in Step 3 gives you average cost-per-request per tier for monitoring and margin alerts. You only need a billing system such as Stripe metered billing when you want to charge customers for that usage, which is a separate concern from watching your own unit economics — and if your margins are eroding, you often want to know weeks before you change what you bill.

Related

Same section:

• Structured Logging with structlog — the full processor pipeline behind Step 1.
• Testing Python APIs with pytest — assert your middleware order and log schema in CI.
• Async Database Access with SQLAlchemy — where a slow trace span usually leads.
• Scaling and Operating Production Python APIs — the parent section overview.

Other sections:

• Designing API Pricing Tiers — turn cost-per-request into defensible margins.
• Logging API Usage Events to Postgres — persist usage for customer-facing reporting.