Controlling LLM API Costs in Production

The decision this page resolves is where you put the ceiling. You can cap tokens per request, cap dollars per customer, cap which model a task may reach, or cap nothing and hope the invoice stays small. Most builders pick the last option by default, then discover on day forty that one trial account burned three months of margin pasting entire PDFs into a free endpoint. Part of the Automating AI Workflows with Python APIs guide, this page picks a specific stack of controls and ships the code for each.

The commercial frame matters more than the technical one. A model call is the only dependency whose unit cost is set by your customer's input rather than by your architecture. A call on a 40,000-token document costs twenty times a call on a 2,000-token one, and the customer paying a flat nineteen dollars a month has no idea they just triggered it. Every control below breaks that link.

The four levers, ranked by return

There are only four things you can do about model spend: send fewer requests, send fewer tokens per request, pay less per token, or refuse to send past a limit. Deduplication and caching attack the first, token budgets the second, model tiering the third, and per-customer spend caps the fourth. Ranked by dollars saved per hour of engineering, that order is almost exactly right — deduplication is a hash lookup, and it routinely removes a quarter of production traffic in workflows where customers re-run the same job.

The figure below shows a realistic stack: a document-summarising background job with roughly 30% exact repeats, a stable 3,000-token system prompt, and a task mix where two thirds of jobs are classification rather than prose generation. Each step is a control you ship, not a modelling trick.

Cumulative effect of four cost controls on model spend A narrowing stack of five bars showing spend falling from an index of 100 to 92 after per-request token budgets, 64 after request deduplication, 50 after prompt caching, and 31 after model tiering. Each control multiplies against the one above it spend index Raw model spend 100 Per-request token budget 92 Dedup on fingerprint 64 Prompt cache reads 50 Model tiering 31 Same workload, same output quality — roughly 31 cents on the original dollar

Budget tokens before you send, not after

The mistake almost everyone makes is measuring cost from the response. By then you have already paid. Token budgeting means deciding, before the request leaves your process, whether this call is allowed to happen at the size it wants. The Anthropic SDK exposes a token counter for exactly this, and it is free to call.

Two numbers govern each call. max_tokens caps the output side and is enforced by the API, so it is your hard ceiling on the expensive half of the bill — output tokens cost five times input tokens on every current model. The input side has no server-side cap, so you count first and either trim the context or reject the job. Rejecting is usually correct in a background worker: a job needing 40,000 tokens of context belongs to a customer who should be on a different plan.

Sequence of a token-budgeted model call A worker calls the token counter, receives an input token count, trims or rejects if over budget, then calls the model API with a capped max_tokens and writes the reported usage to a cost ledger. Count first, spend second Worker Token counter Model API count_tokens(prompt) input_tokens = 14,200 Over 12,000: trim or reject messages.create(max_tokens=1024) usage.input_tokens to ledger The counter call is free; the call you avoid is the one you would have paid for
Python
import os
from dataclasses import dataclass

import anthropic

client = anthropic.AsyncAnthropic()

MODEL = os.getenv("LLM_MODEL", "claude-haiku-4-5")
MAX_INPUT_TOKENS = int(os.getenv("LLM_MAX_INPUT_TOKENS", "12000"))
MAX_OUTPUT_TOKENS = int(os.getenv("LLM_MAX_OUTPUT_TOKENS", "1024"))
INPUT_PER_MTOK = float(os.getenv("LLM_INPUT_PER_MTOK", "1.00"))
OUTPUT_PER_MTOK = float(os.getenv("LLM_OUTPUT_PER_MTOK", "5.00"))


class BudgetExceeded(RuntimeError):
    """Raised before any billable request is sent."""


@dataclass(frozen=True, slots=True)
class CallResult:
    text: str
    input_tokens: int
    output_tokens: int

    @property
    def usd(self) -> float:
        return (
            self.input_tokens * INPUT_PER_MTOK
            + self.output_tokens * OUTPUT_PER_MTOK
        ) / 1_000_000


async def budgeted_call(system: str, user_text: str) -> CallResult:
    messages = [{"role": "user", "content": user_text}]
    counted = await client.messages.count_tokens(
        model=MODEL, system=system, messages=messages
    )
    if counted.input_tokens > MAX_INPUT_TOKENS:
        raise BudgetExceeded(
            f"{counted.input_tokens} input tokens exceeds {MAX_INPUT_TOKENS}"
        )

    message = await client.messages.create(
        model=MODEL,
        system=system,
        messages=messages,
        max_tokens=MAX_OUTPUT_TOKENS,
    )
    return CallResult(
        text="".join(b.text for b in message.content if b.type == "text"),
        input_tokens=message.usage.input_tokens,
        output_tokens=message.usage.output_tokens,
    )

Read the price constants from the environment rather than hardcoding them: vendor pricing changes, and a ledger that silently reports last year's rates is worse than no ledger. The reasoning behind calculating cost per API request for your own endpoints applies doubly here, because the input size is not yours to control.

Caching and deduplication: two different wins

These get conflated constantly. Deduplication means you never call the model, because you already hold the answer for this exact input. Prompt caching means you do call it, but the shared prefix bills at roughly a tenth of the normal input rate. Ship both — they compose.

Deduplication needs a fingerprint covering everything that changes the answer: model ID, prompt version, system text, and the normalised user input. Leave the prompt version out and your first prompt improvement silently serves stale results to every returning customer. Store the result in Redis with a TTL tied to how long the answer stays true; the trade-offs match those in cache invalidation strategies.

Python
import hashlib
import json
import os

import redis.asyncio as redis

r = redis.from_url(os.getenv("REDIS_URL", "redis://localhost:6379/0"))
PROMPT_VERSION = os.getenv("LLM_PROMPT_VERSION", "v3")
DEDUP_TTL = int(os.getenv("LLM_DEDUP_TTL_SECONDS", "86400"))


def fingerprint(system: str, user_text: str) -> str:
    digest = hashlib.sha256()
    for part in (MODEL, PROMPT_VERSION, system, user_text.strip()):
        digest.update(part.encode("utf-8"))
        digest.update(b"\x00")
    return f"llm:{digest.hexdigest()}"


async def cached_call(system: str, user_text: str) -> CallResult:
    key = fingerprint(system, user_text)
    if hit := await r.get(key):
        payload = json.loads(hit)
        return CallResult(payload["text"], 0, 0)  # zero cost, zero tokens

    result = await budgeted_call(system, user_text)
    await r.set(key, json.dumps({"text": result.text}), ex=DEDUP_TTL)
    return result

Prompt caching is a separate switch on the request itself. Mark the stable prefix — system prompt, few-shot examples, the schema you always send — with a cache breakpoint, and keep every volatile byte after it. A timestamp interpolated into the system prompt invalidates the whole prefix on every call, which is the usual reason builders report that caching "does nothing".

Cost per million input tokens under three caching outcomes Bar chart on a five-dollar-per-million base rate: uncached input costs five dollars, a cache write costs six dollars twenty-five, and a cache read costs fifty cents. Dollars per 1M input tokens at a $5.00 base rate $5.00 Uncached every call $6.25 Cache write first call only $0.50 Cache read every call after Two calls on the same prefix already beat two uncached calls

Model tiering: route down, never up

Price spreads across the tiers are wide enough that routing is the largest structural saving available. On a representative job of 2,000 input tokens and 400 output tokens, the small model costs $0.004, the mid tier $0.012, and the frontier tier $0.020. At 50,000 jobs a month that is $200 against $1,000 — the difference between a healthy gross margin and an apology to your accountant.

Route down by default and promote a task only when a real evaluation shows the cheap tier failing. Classification, extraction, tagging, and short structured summaries almost never need a frontier model; multi-step reasoning, long tool chains, and anything a customer reads verbatim usually do. Encode that as data, not scattered conditionals.

Decision tree routing a job to a model tier An incoming job branches on three conditions — fixed schema output, customer-facing prose, or multi-step reasoning — routing respectively to a small, mid, and frontier model with their per-million token prices. Pick the cheapest tier that passes your eval Incoming job Fixed schema out? classify / extract Customer reads it? drafts / summaries Multi-step work? tool chains Small tier $1 in / $5 out Mid tier $3 in / $15 out Frontier tier $5 in / $25 out Promote a task only after the cheap tier measurably fails
Python
import os
from dataclasses import dataclass


@dataclass(frozen=True, slots=True)
class Tier:
    model: str
    input_per_mtok: float
    output_per_mtok: float
    max_tokens: int


TIERS = {
    "small": Tier(os.getenv("LLM_TIER_SMALL", "claude-haiku-4-5"), 1.00, 5.00, 512),
    "mid": Tier(os.getenv("LLM_TIER_MID", "claude-sonnet-5"), 3.00, 15.00, 2048),
    "frontier": Tier(os.getenv("LLM_TIER_TOP", "claude-opus-4-8"), 5.00, 25.00, 4096),
}


def route(task: str, input_tokens: int) -> Tier:
    match task:
        case "classify" | "extract" | "tag" | "route":
            return TIERS["small"]
        case "summarize" if input_tokens < 8_000:
            return TIERS["small"]
        case "summarize" | "draft" | "rewrite":
            return TIERS["mid"]
        case _:
            return TIERS["frontier"]

A table keeps the policy reviewable by whoever owns the margin, which on a side project is you at midnight.

Task shapeTierWhy
Classify, tag, extractSmallOutput is a schema, not prose
Short summary under 8kSmallQuality gap is invisible
Customer-facing draftMidTone matters, volume is lower
Multi-step or tool chainFrontierErrors compound across steps

Per-customer spend caps

Token budgets protect you from one bad request. Spend caps protect you from one bad customer. Without them, a single account on a flat plan consumes the margin of a hundred others and you find out from the invoice. The pattern is a Redis counter per customer per billing period, incremented atomically before the call with an estimate and reconciled afterwards with actual usage.

Reserve before you spend, not after. If you increment only on completion, a burst of fifty concurrent jobs all read the same under-cap value and all proceed. A Lua script makes check-and-increment one atomic operation, which is the difference between a cap and a suggestion. Pair the ledger with durable records by logging API usage events to Postgres, so billing disputes have an audit trail Redis cannot give you.

State machine for a per-customer spend cap A customer moves from under cap to a soft cap at eighty percent where requests are tiered down, then to a hard cap where requests are refused, and finally back to under cap when the billing period resets. One customer, one ledger, three states Under cap serve normally Soft cap warn, tier down Hard cap refuse, upsell Cycle reset ledger cleared 80% 100% month end A new billing period clears the counter and re-opens the account Reserve the estimate before the call; reconcile the actual after it
Python
import os

RESERVE_SCRIPT = """
local spent = tonumber(redis.call('GET', KEYS[1]) or '0')
local cap = tonumber(ARGV[1])
local estimate = tonumber(ARGV[2])
if spent + estimate > cap then
  return -1
end
redis.call('INCRBYFLOAT', KEYS[1], ARGV[2])
redis.call('EXPIREAT', KEYS[1], ARGV[3])
return 1
"""

DEFAULT_CAP_USD = float(os.getenv("LLM_CUSTOMER_CAP_USD", "5.00"))


class SpendCapReached(RuntimeError):
    """The customer has used their allowance for this billing period."""


async def reserve(customer_id: str, period: str, estimate_usd: float,
                  period_end_epoch: int, cap_usd: float = DEFAULT_CAP_USD) -> None:
    script = r.register_script(RESERVE_SCRIPT)
    allowed = await script(
        keys=[f"spend:{customer_id}:{period}"],
        args=[cap_usd, estimate_usd, period_end_epoch],
    )
    if allowed == -1:
        raise SpendCapReached(customer_id)


async def settle(customer_id: str, period: str, estimate_usd: float,
                 actual_usd: float) -> None:
    delta = actual_usd - estimate_usd
    if delta:
        await r.incrbyfloat(f"spend:{customer_id}:{period}", delta)

Set the estimate slightly high: over-reserving costs a customer a few cents of headroom, while under-reserving costs you real money on every concurrent burst. If your plans already meter usage through Stripe metered billing configuration, the same period key feeds both the cap and the invoice.

Alerting on cost regressions

The metric to alert on is not total spend, which rises when you grow — the outcome you want. Alert on cost per successful job, computed on a rolling window against the previous one. A prompt edit that adds 800 tokens of instructions, a retry loop that silently doubles calls, or a routing bug that sends classification work to the frontier tier all show up there within minutes and nowhere else until the invoice.

Emit one structured log line per model call carrying customer ID, task, tier, both token counts, and the dollar figure, then aggregate. Structured logging with structlog makes those lines queryable without regex archaeology. Set the threshold at a 25% rise in median cost per job over the trailing hour against the same hour a day earlier — tight enough to catch a bad deploy, loose enough to survive one customer with long documents.

When to use each control, and when not to

At the prototype stage, ship token budgets and nothing else. They take twenty minutes and stop the catastrophic case. Deduplication earns its keep once you have repeat traffic — below a few thousand jobs a month, the Redis instance costs more than the calls it saves. Prompt caching pays from the second identical-prefix request onward, so turn it on as soon as your system prompt stabilises above about a thousand tokens; below that, most providers will not cache the prefix at all.

Model tiering is worth the routing table once your monthly model bill passes the cost of a day of your own time. Per-customer spend caps become mandatory the moment you sell a flat-rate plan with model calls behind it. Skip them if you charge purely per token with a healthy markup — you have no exposure, and the enforcement code is pure complexity. The exception is free tiers, where caps double as the mechanism for preventing free-tier abuse.

Migration path from an uncontrolled service

Adding all of these at once to a running service is how you ship an outage. Do it in this order, one deploy each.

  1. Add the cost ledger in observe-only mode. Log tokens and dollars per call for a week before changing any behaviour.
  2. Add max_tokens and the input count check, but log violations instead of raising. The log shows where a real cap would fire, and on whose traffic.
  3. Flip the input check to raising, with the limit just above the 99th percentile you measured. The tail that breaks was always the expensive tail.
  4. Introduce the fingerprint cache with a one-hour TTL to prove the hit rate, then extend it once you trust the fingerprint.
  5. Add the routing table, moving one task type down a tier at a time and diffing outputs against saved samples before the next move.
  6. Add spend caps last: a warning-only soft cap first, then refusal once you have a week of real distributions.

Builder verdict

If you ship only two of these, ship the per-request token budget and the fingerprint cache. Together they take an afternoon, need no infrastructure beyond the Redis you already run for caching Python API responses with Redis, and they remove both the catastrophic tail and the dumbest recurring spend. Model tiering saves more in absolute dollars, but it costs you an evaluation harness and ongoing judgement calls about quality — real engineering time you might rather spend on features that grow revenue. Per-customer spend caps stop being optional the day you sell a flat-rate plan, not because they save the most but because they are the only thing between one enthusiastic customer and a month of negative margin. Build the ledger first: every other control is guesswork until you know what a job costs.

FAQ

How much of my revenue should model spend be? Keep model spend under 25% of the revenue from the feature it powers, and treat 40% as the line where the product needs repricing rather than more optimisation. On a nineteen-dollar plan that is roughly five dollars of calls a month — about twelve hundred typical jobs on the small tier. If many accounts run past that, the fix is a usage-based plan, not a cleverer cache.

Will a cheaper model cost me customers? Only if you route by guess instead of by evaluation. Save fifty real production inputs with their frontier-tier outputs, run the cheap tier on the same inputs, and diff. On classification and extraction the results are usually indistinguishable, and the tasks where they are not become obvious immediately. Never testing is the worse risk: you pay five times over for quality nobody perceives.

What happens to caching when I change my prompt? Every answer keyed on the old prompt version is orphaned, which is what you want — bump the version constant in the fingerprint and old entries expire on their own TTL. Provider-side caching also resets, so the first request after a deploy pays the cache-write premium of about 1.25 times the normal input rate. Budget one cold hour and never ship prompt edits at your traffic peak.

Do spend caps hurt conversion? A hard cap with no warning does. A soft cap that emails the customer at 80%, quietly routes them to a cheaper tier, and offers a one-click upgrade converts better than an unlimited plan you silently rate-limit. Treat the cap as a pricing surface, not a defence mechanism — the customers who hit it get the most value and are your easiest upsell.

How do I estimate cost before the job runs, for a reservation? Count the input tokens exactly, then assume the output hits your configured max_tokens. That over-reserves on most calls and reconciles down at settlement, which is the safe direction. For streaming endpoints, where output length is unknown until the stream closes, reserve on max_tokens up front and settle when the final usage arrives — streaming LLM responses through FastAPI covers where to hook that settlement.