How to Use Python Requests for Beginners: A Builder's Guide to APIs

The requests library is still the fastest way to get a paid integration working, and it is still the fastest way to take down your own product at 3am if you use it the way most tutorials teach it. This walkthrough sits under the Understanding REST vs GraphQL guide, because nearly every beginner request you write targets a REST endpoint, and the shape of that endpoint decides how many calls you pay for.

Everything here is the production form from the start: environment-driven configuration, a reused connection pool, explicit timeouts, a retry policy that only fires on the status codes worth retrying, and a clear number for when the synchronous model stops paying for itself. If you want the wider tour of HTTP clients afterwards, the making HTTP requests with the requests library guide covers the surrounding territory.

Key takeaways:

  • Reuse one Session per upstream service — it roughly triples throughput for free
  • Always pass a (connect, read) timeout tuple, never a bare number and never nothing
  • Retry 429, 502, 503 and 504; never blind-retry a 400 or a 422
  • Move to httpx when concurrency, not code style, becomes the bottleneck

Install and Configure Without Hardcoding Anything

Install into a virtual environment and pin the minor version. requests is stable, but urllib3 2.x changed retry semantics under it, so an unpinned install can silently alter your backoff behaviour on a redeploy.

Bash
python -m venv .venv && . .venv/bin/activate
pip install "requests>=2.32,<3"

Configuration is where beginner code accumulates its worst debt. A base URL typed inline becomes forty base URLs typed inline, and then staging traffic hits production. Read every endpoint, key and tuning value from the environment, and give each one a sane default so a fresh clone still boots.

Python
import os
from dataclasses import dataclass

@dataclass(frozen=True, slots=True)
class ApiConfig:
    base_url: str
    api_key: str
    connect_timeout: float
    read_timeout: float

    @classmethod
    def from_env(cls) -> "ApiConfig":
        return cls(
            base_url=os.getenv("ITEMS_API_BASE_URL", "https://api.example.com/v1"),
            api_key=os.getenv("ITEMS_API_KEY", ""),
            connect_timeout=float(os.getenv("ITEMS_API_CONNECT_TIMEOUT", "3.05")),
            read_timeout=float(os.getenv("ITEMS_API_READ_TIMEOUT", "27")),
        )

config = ApiConfig.from_env()

The odd-looking 3.05 connect timeout is deliberate. TCP retransmits its initial SYN on a three-second window, so a connect timeout just above a multiple of three gives the kernel one full retry before you give up. A flat timeout=3 throws away roughly a third of your recoverable connection attempts on a lossy network. Storing the key in the environment is the minimum bar; when you have real customers, read rotating API keys without downtime before your first credential leak forces the question.

Use One Session, Not Bare Module Calls

requests.get() creates a brand new Session, opens a TCP connection, performs a full TLS handshake, sends one request, and throws the connection away. On a typical cloud-to-cloud hop that handshake costs 80–100 ms against 40 ms of actual server work. You are paying more than twice as much for the greeting as for the conversation.

A Session keeps the connection alive and reuses it. The second and third calls skip DNS, the TCP three-way handshake and the TLS negotiation entirely.

Wall-clock timeline for three calls with and without a Session Without a Session each of three calls pays a ninety millisecond handshake plus forty milliseconds of server work, totalling 390 milliseconds. With a Session only the first call pays the handshake, totalling 210 milliseconds. Three sequential calls to the same host requests.get() 390 ms Session 210 ms DNS + TCP + TLS, 90 ms server work, 40 ms

Build the session once, at import time or in a FastAPI lifespan handler, and attach the headers every call needs.

Python
import requests

def build_session(cfg: ApiConfig) -> requests.Session:
    session = requests.Session()
    session.headers.update({
        "Authorization": f"Bearer {cfg.api_key}",
        "Accept": "application/json",
        "User-Agent": os.getenv("API_USER_AGENT", "builder-app/1.0"),
    })
    return session

session = build_session(config)

Set a real User-Agent. When an upstream provider starts throttling you, the first thing their support team asks for is the identifier in your requests, and python-requests/2.32 is not one.

Execute Your First GET Request

Pass query parameters through params and let requests handle percent-encoding — hand-built query strings break the first time a customer name contains an ampersand. Never omit the timeout.

Python
def fetch_items(category: str, limit: int = 10) -> dict:
    response = session.get(
        f"{config.base_url}/items",
        params={"category": category, "limit": limit},
        timeout=(config.connect_timeout, config.read_timeout),
    )
    response.raise_for_status()
    return response.json()
Anatomy of a GET request and its response The URL, session headers and params dictionary combine into one GET call; the server returns a status line, response headers and a JSON body. base_url + /items session headers params, url-encoded GET upstream API 40 ms of work status_code headers .json() body raise_for_status() inspects only the status code, never the body

That last line is the trap. Plenty of APIs answer 200 OK with {"error": "quota exceeded"} in the body, and raise_for_status() waves it straight through. Read the payload as well as the status; the debugging 401 Unauthorized API errors walkthrough shows how far apart the status code and the real cause can drift.

Submit Data via POST Requests

Pass a dictionary to json= and requests serialises it and sets Content-Type: application/json for you. Use data= only for form-encoded submissions. Mixing the two is the single most common cause of a 400 on a first integration, because data={"a": 1} sends a=1 on the wire, and a JSON API has no idea what that is.

Python
import uuid

def create_user(name: str, tier: str, idempotency_key: str | None = None) -> dict:
    # Generate the key ONCE per logical operation, not once per attempt.
    response = session.post(
        f"{config.base_url}/users",
        json={"name": name, "tier": tier},
        timeout=(config.connect_timeout, config.read_timeout),
        headers={"Idempotency-Key": idempotency_key or str(uuid.uuid4())},
    )
    response.raise_for_status()
    return response.json()
json equals versus data equals in requests A four-row matrix comparing the content type, wire format, nested object handling and typical failure of the json and data keyword arguments. json=payload data=payload Content-Type application/json x-www-form-urlencoded Bytes on the wire {"tier": "pro"} tier=pro Nested objects preserved exactly flattened or dropped Typical failure none 400 on a JSON endpoint

Two edge cases bite quickly. First, a datetime is not JSON-serialisable, so json= raises TypeError before a byte leaves your process — convert with .isoformat() at the boundary. Second, any POST that creates a billable object deserves an idempotency key, because a read timeout does not tell you whether the server acted. Send the same key on the retry and a well-built API returns the original result instead of charging twice. The building an idempotent webhook receiver page shows the same guarantee from the receiving side.

Parse JSON and Extract Values Safely

Use response.json() rather than json.loads(response.text). It handles the character encoding declared in the response headers and raises a requests.exceptions.JSONDecodeError you can catch specifically — which matters when an upstream outage returns an HTML error page with a 200 status.

Python
import requests

def read_item(item_id: str) -> tuple[str, float]:
    response = session.get(
        f"{config.base_url}/items/{item_id}",
        timeout=(config.connect_timeout, config.read_timeout),
    )
    response.raise_for_status()
    try:
        data = response.json()
    except requests.exceptions.JSONDecodeError:
        raise RuntimeError(f"non-JSON body from {response.url}: {response.text[:200]}")
    return data.get("id", "unknown"), float(data.get("price", 0.0))

Defensive .get() calls keep a schema change from crashing your worker, but they also let bad data flow silently into your billing tables. Once money depends on a field, stop guessing and validate: the validating JSON with Pydantic v2 page covers the model layer, and the wider parsing JSON responses guide covers the rest. If a single response ever exceeds a few megabytes, response.json() will hold the raw bytes, the decoded string and the parsed object in memory at once — roughly three times the payload — so switch to the approach in handling large JSON payloads with streaming before your 512 MB container starts getting OOM-killed.

Retry the Right Failures and Only Those

Blanket retries are worse than none. Retrying a 400 burns quota to receive the same rejection three times; retrying a non-idempotent POST can double-charge a customer. Sort responses into four buckets and treat each differently.

Decision tree for handling a response by status class A response splits four ways: 2xx parses and returns, 429 and 503 wait for the Retry-After header, 500 502 and 504 use capped exponential backoff, and other 4xx codes raise without retrying. response arrives 2xx 429 or 503 500, 502, 504 other 4xx parse and return obey Retry-After backoff, cap at 3 raise, no retry Auto-retry idempotent verbs only: GET, HEAD, PUT, DELETE

Wire the transport-level policy into an adapter so it applies to every call the session makes, then keep a match statement for the decisions your application still owns.

Python
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

def mount_retries(session: requests.Session, cfg: ApiConfig) -> None:
    retry = Retry(
        total=int(os.getenv("ITEMS_API_MAX_RETRIES", "3")),
        backoff_factor=float(os.getenv("ITEMS_API_BACKOFF", "0.5")),
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=frozenset({"GET", "HEAD", "PUT", "DELETE"}),
        respect_retry_after_header=True,
        raise_on_status=False,
    )
    adapter = HTTPAdapter(
        max_retries=retry,
        pool_connections=int(os.getenv("ITEMS_API_POOL", "16")),
        pool_maxsize=int(os.getenv("ITEMS_API_POOL", "16")),
    )
    session.mount("https://", adapter)

mount_retries(session, config)

def classify(response: requests.Response) -> str:
    match response.status_code:
        case code if 200 <= code < 300:
            return "ok"
        case 401 | 403:
            return "credentials"
        case 429:
            return "throttled"
        case code if code >= 500:
            return "upstream"
        case _:
            return "client_error"

With backoff_factor=0.5 urllib3 sleeps roughly 0.5s, 1s then 2s — about 3.5 seconds of worst-case added latency, which is fine for a background job and far too long inside a request a customer is waiting on. Size pool_maxsize to your thread count too; the default of 10 makes a 32-thread pool log Connection pool is full, discarding connection and silently drop back to handshaking per call, quietly erasing the gain from the previous section. For application-level policy across several services, retrying failed HTTP requests with tenacity gives you more control than the adapter, and debugging 429 Too Many Requests errors covers what to do when the throttling never lets up.

What the Synchronous Model Actually Costs

Here is the number that decides your architecture. One synchronous worker against a 40 ms endpoint moves about 7.7 requests per second with fresh connections, and about 22 with a warm session. Add 16 threads and a matched pool and you reach roughly 210 per second before the GIL and context switching flatten the curve. The same workload on httpx with 50 concurrent tasks in one event loop reaches around 640.

Throughput by client strategy against a forty millisecond endpoint Bare requests calls reach 7.7 requests per second, a warm Session 22, a Session with 16 threads 210, and async httpx with 50 concurrent tasks 640. Requests per second, one 2 vCPU container bare requests.get 7.7/s warm Session 22/s Session + 16 threads 210/s httpx, 50 tasks 640/s 40 ms upstream latency, 90 ms handshake when not reused

Translate that into money. A job that fans out 500,000 upstream calls a month finishes in about 40 minutes of wall clock at 210 per second, versus over six hours at 22. On a container billed at roughly 5 cents an hour that is a difference of about 30 cents a month — nothing. The real cost is the six hours of latency in your data freshness and the retry storm when a run overruns its schedule window. Work the same arithmetic for your own endpoints using calculating cost per API request, and if the workload is polling for changes rather than fetching known resources, check whether webhooks beat polling before you optimise the client at all.

My rule: stay on requests for scripts, cron jobs, webhook handlers and anything under about 50 concurrent outbound calls. Switch to httpx when you are already inside an async framework, because a synchronous call inside an async route blocks the entire event loop and destroys the concurrency you paid for. The httpx vs requests for async comparison has the migration detail, and mocking external APIs with respx covers testing once you land there.

Common Mistakes to Avoid

  • No timeout. The default is infinite. One hung upstream connection pins a worker forever and your health check still reports green.
  • A new Session per call. Triples your latency and exhausts ephemeral ports under load.
  • data= where the API wants json=. A 400 with no useful message, every time.
  • Trusting raise_for_status() alone. Errors returned as 200 with an error body sail straight past it.
  • Logging the whole response. Bearer tokens and customer emails end up in your log store; log the status, URL and request id instead, ideally through structured logging with structlog.
  • Ignoring pool_maxsize. Threads without pool headroom quietly fall back to a handshake per request.

FAQ

What does a requests-based integration cost to run at one million calls a month? About one 2 vCPU container running a few hours a day. At 210 requests per second with a threaded session, a million calls take roughly 80 minutes of compute, which is under a dollar on any container host. Your bill is set by the upstream provider's per-call pricing, not by Python.

Is requests fast enough for a paid API I resell? Yes, up to a point. Inside a synchronous worker it is fine to a few hundred concurrent outbound calls. If your own API is async and you call requests from a route handler, you block the event loop and your p99 collapses — that is the moment to move to httpx, not before.

How do I rotate an API key without redeploying every worker? Read the key at request time from a config object you can refresh, not once at import. Support two valid keys during the overlap window so in-flight workers keep succeeding, then retire the old one. The full pattern is in rotating API keys without downtime.

Will migrating from requests to httpx break my code later? Barely. The sync API is close enough that most call sites change only the client object, and timeout, params and json= keep their meanings. The real work is converting your call sites to await and auditing anything that relied on thread-local state, so budget a day per service rather than a week.

Should I retry a failed POST that charges a customer? Only with an idempotency key the upstream honours. Without one, a read timeout leaves you unable to tell a failed charge from a successful one, and a retry can bill twice. If the provider has no idempotency support, reconcile by listing recent objects before creating a new one.