Respecting robots.txt and API Terms of Service

Every data-backed side hustle eventually hits the same fork: the numbers you need are sitting on someone else's server, and you have to decide whether to take them with a crawler or pay for them through an official endpoint. This page resolves that fork with code rather than hand-waving — how to parse robots.txt properly in Python, how to honour crawl-delay without wrecking your pipeline throughput, which four clauses in a terms document actually decide the question, and the point at which scraping stops being defensible and the official API becomes the only sane call. It belongs to the Web Scraping vs Official APIs guide, which argues the broader case that official endpoints usually win on total cost.

Get the framing right first. robots.txt is not a law and it is not a security control — it is a machine-readable request from the site operator, and ignoring it is the single fastest way to turn a technical dispute into a legal one. The terms of service, by contrast, are a contract you accept the moment you create an account or an API key. Breaking robots.txt gets you blocked. Breaking terms gets your account terminated, your key revoked, and in the worst case gets your company named in a letter. Those are different risks with different price tags, and you should price both before you write the first line of a crawler.

The decision, in order

Work the question top down, not bottom up. Most builders start by asking "can I scrape this page?" when the first question is "does this vendor already sell me the data?" Answer them in the wrong order and you spend three weeks building a fragile crawler that a $29/month plan would have replaced on day one.

Decision order before fetching third-party data Check for an official API first, then whether robots.txt allows the path, then whether the terms of service permit the intended use, and only then scrape at the published crawl delay. Official API exists? yes Use the API no Path allowed by robots? no Stop, do not fetch yes Terms permit this use? no Licence it or stop yes Fetch at the crawl delay cache every response

Each downward step costs you more engineering and carries more risk than the one above it. That asymmetry is the whole argument: the cheapest path is almost always the highest branch you can reach.

Parsing robots.txt properly in Python

The standard library ships urllib.robotparser, and it is fine for simple cases, but it is strict about the original 1994 draft and misses modern directives that real sites use. Protego — the parser Scrapy uses — implements the Google-flavoured rules including wildcards, Allow precedence by specificity, and Crawl-delay. Install it with pip install protego httpx and fetch the file yourself so you control timeouts and caching.

Python
import os
import asyncio
from dataclasses import dataclass
from urllib.parse import urlsplit, urlunsplit

import httpx
from protego import Protego

USER_AGENT = os.getenv("CRAWLER_USER_AGENT", "builder-bot/1.0 (+https://example.invalid/bot)")
FETCH_TIMEOUT_S = float(os.getenv("CRAWLER_FETCH_TIMEOUT_S", "10"))
DEFAULT_DELAY_S = float(os.getenv("CRAWLER_DEFAULT_DELAY_S", "5"))


@dataclass(slots=True)
class HostPolicy:
    rules: Protego | None
    delay_s: float

    def allows(self, url: str) -> bool:
        # No robots.txt at all means no restrictions were published.
        return True if self.rules is None else self.rules.can_fetch(url, USER_AGENT)


def robots_url_for(url: str) -> str:
    parts = urlsplit(url)
    return urlunsplit((parts.scheme, parts.netloc, "/robots.txt", "", ""))


async def load_policy(client: httpx.AsyncClient, url: str) -> HostPolicy:
    try:
        resp = await client.get(robots_url_for(url), timeout=FETCH_TIMEOUT_S)
    except httpx.HTTPError:
        # Network failure is not permission. Assume the strictest sane default.
        return HostPolicy(rules=None, delay_s=DEFAULT_DELAY_S)

    match resp.status_code:
        case 200:
            rules = Protego.parse(resp.text)
            delay = rules.crawl_delay(USER_AGENT)
            return HostPolicy(rules=rules, delay_s=float(delay or DEFAULT_DELAY_S))
        case 404 | 410:
            return HostPolicy(rules=None, delay_s=DEFAULT_DELAY_S)
        case 401 | 403:
            # A protected robots.txt means the whole site is off limits.
            return HostPolicy(rules=Protego.parse("User-agent: *\nDisallow: /"), delay_s=DEFAULT_DELAY_S)
        case _:
            return HostPolicy(rules=None, delay_s=DEFAULT_DELAY_S)


async def main() -> None:
    target = os.environ["CRAWLER_TARGET_URL"]
    async with httpx.AsyncClient(headers={"user-agent": USER_AGENT}) as client:
        policy = await load_policy(client, target)
        print(f"allowed={policy.allows(target)} delay={policy.delay_s}s")


if __name__ == "__main__":
    asyncio.run(main())

Three details matter more than the parsing itself. First, a 401 or 403 on robots.txt means the entire site is disallowed under the widely followed convention — do not treat a locked file as a green light. Second, a network error is not consent; fall back to your most conservative default instead of assuming an empty ruleset. Third, cache the parsed policy per host for at least an hour so a thousand-page crawl fetches robots.txt once, not a thousand times. If you already run Redis for caching API responses, park the raw file body there with a TTL and skip the refetch entirely.

If you would rather stay dependency-free, urllib.robotparser.RobotFileParser exposes can_fetch(useragent, url), crawl_delay(useragent) and site_maps() against text you feed to parse(lines). It is a reasonable choice for a single well-behaved target; it is the wrong choice the moment your crawler faces wildcard rules.

Honouring crawl-delay without killing throughput

Crawl-delay: 10 means ten seconds between your requests to that host. Builders routinely treat this as a global sleep and then wonder why a nightly job takes six hours. The fix is to make the delay per host, not per process, and to run hosts concurrently. The chart below uses real arithmetic: one host, one worker, and whatever delay the site publishes.

Pages fetched per hour per host at each crawl delay A bar chart comparing sequential throughput for one host: 7200 pages per hour at a half-second delay, 1800 at two seconds, 720 at five seconds, and 360 at ten seconds. Pages per hour, one host, one worker 0.5s delay 7,200 2s delay 1,800 5s delay 720 10s delay 360 Ten hosts in parallel multiply every bar by ten.

Ten seconds per host caps you at 360 pages an hour. Crawl fifty hosts concurrently and that same policy yields 18,000 pages an hour while every individual site sees one polite request every ten seconds. This gate makes that shape trivial to implement.

Python
import os
import time
import asyncio
from collections import defaultdict

JITTER_RATIO = float(os.getenv("CRAWLER_JITTER_RATIO", "0.25"))


class HostGate:
    """Serialises requests per host and spaces them by the published delay."""

    def __init__(self) -> None:
        self._locks: dict[str, asyncio.Lock] = defaultdict(asyncio.Lock)
        self._next_ok: dict[str, float] = {}

    async def acquire(self, host: str, delay_s: float) -> None:
        async with self._locks[host]:
            now = time.monotonic()
            wait = self._next_ok.get(host, 0.0) - now
            if wait > 0:
                await asyncio.sleep(wait)
            spacing = delay_s * (1.0 + JITTER_RATIO)
            self._next_ok[host] = time.monotonic() + spacing


gate = HostGate()


async def polite_get(client, url: str, host: str, delay_s: float):
    await gate.acquire(host, delay_s)
    resp = await client.get(url)
    if resp.status_code == 429:
        retry_after = float(resp.headers.get("retry-after", delay_s * 4))
        await asyncio.sleep(retry_after)
    return resp

The Retry-After branch is not optional. A 429 is the server telling you its own limit is tighter than what robots.txt advertised, and the correct response is to widen your spacing for that host permanently, not just for one retry. The same backoff discipline applies to paid endpoints — see debugging 429 Too Many Requests errors for the client-side patterns, and retrying failed HTTP requests with tenacity if you want the retry policy declared rather than hand-rolled.

The four clauses that decide the question

robots.txt governs fetching. Terms of service govern what you may do with what you fetched, and that is where commercial projects get hurt. Read four things before you build, and read them for the API too — official access is not a blanket permission slip.

Terms of service clause matrix Four clause types with their permissive wording on one side and their blocking wording on the other: commercial use, caching and storage, volume limits, and redistribution. Clause Green light wording Blocking wording Commercial use "any lawful purpose" "personal, non-commercial" Caching "cache for up to 30 days" "no local copy or database" Volume "published quota, paid tiers" "excessive use, sole judgement" Redistribution "display with attribution" "no resale or sublicensing"

Commercial use is the clause that kills side hustles. Plenty of free tiers permit "personal and non-commercial" use only, which means the moment you charge a customer you are in breach even though nothing technical changed. Caching and storage decides your architecture: a "no persistent storage" clause forbids the very database that makes your product fast, and a thirty-day cache window sets your refresh schedule for you. Volume language written as "excessive use as determined by us" is a rate limit with no number, so budget for sudden throttling. Redistribution decides your business model — if you may display but not resell, a paid API wrapper around that data is not a product you can legally sell.

Write those four answers into a short sources.toml in your repo and load it with tomllib, so the constraint lives next to the code that has to obey it rather than in someone's memory.

Python
import os
import tomllib
from pathlib import Path

SOURCES_FILE = Path(os.getenv("SOURCES_FILE", "sources.toml"))

with SOURCES_FILE.open("rb") as fh:
    SOURCES = tomllib.load(fh)


def cache_ttl_seconds(source: str) -> int:
    policy = SOURCES["source"][source]
    if not policy.get("commercial_use_allowed", False):
        raise PermissionError(f"{source} forbids commercial use — do not ship it")
    return int(policy.get("cache_days", 1)) * 86_400

That raise is deliberate. A source whose terms forbid commercial use should fail loudly at startup, not quietly earn you revenue for four months until someone notices.

When to choose the official API instead

Pick the API — even a paid one — the moment any of these is true. At pre-revenue stage, use the free tier and skip the crawler entirely; your scarcest resource is shipping velocity, and a crawler costs a week of build plus permanent maintenance. At any traffic level above a few thousand records a day, the delay arithmetic above bites: 10-second politeness caps a single host at 8,640 pages a day, and beating that legitimately means paying. On cost model, run the real numbers — a scraper that needs residential proxies at roughly $3–8 per gigabyte, a headless browser fleet, and a developer babysitting selector drift routinely lands north of $300 a month in blended cost, which buys a mid-tier API plan outright. Work it through with calculating cost per API request before you commit.

Scraping stays defensible in a narrow band: public, non-authenticated pages, allowed by robots.txt, terms that do not forbid your use, low volume, and no official product that sells the same data. Outside that band, the crawler is a liability with a cron schedule. And if the page is JavaScript-rendered, remember that scraping with Playwright and Python multiplies both your compute bill and your fragility.

Migrating from crawler to official API

Do not rip the scraper out. Put both behind one interface, then switch by environment variable — the whole migration becomes a config change plus a backfill.

Adapter that lets a scraper and an API swap places A scraper source and an API source both satisfy one fetch protocol, which feeds the normaliser and store, selected at runtime by an environment variable. ScraperSource robots + delay ApiSource key + quota FeedSource protocol fetch() -> list[Record] Normalise + store unchanged downstream selected by FEED_SOURCE env var — swap without touching the pipeline

The concrete steps are short. Define a Protocol with a single fetch() coroutine returning your normalised record type. Wrap the existing crawler in a class that satisfies it and confirm the pipeline still passes its tests. Write the API implementation against the same protocol, reading the key from os.environ as covered in handling API authentication in Python. Run both in shadow mode for a week and diff the outputs to find schema gaps. Flip FEED_SOURCE to api, keep the scraper in the repository for one more release as a fallback, then delete it. If your fetch runs on a schedule, the switch is a single env change in the job definition covered by scheduling data pipelines with cron.

Builder verdict

The official API wins, and it is not close for anyone building something they intend to charge for. A paid endpoint gives you a contract, a versioned schema, a support channel, and a legal position you can explain to a customer or an acquirer — for a monthly price you can put in a spreadsheet. A crawler gives you an unpriced liability: proxy bills that scale with volume, selector drift that breaks silently on a Saturday, and terms exposure that only surfaces when you are successful enough to be noticed. Scrape when there is genuinely no API, the data is public, robots.txt allows it, and the terms do not forbid your use — and even then, build it behind an adapter, honour the crawl delay, cache everything, and treat the crawler as a temporary bridge rather than infrastructure. The version of you that ships in three days on a documented endpoint beats the version that spends three weeks fighting Cloudflare, every single time.

FAQ

Does ignoring robots.txt actually cost me money, or is it just etiquette? It costs money. The immediate bill is IP bans and CAPTCHA-solving services, which push a working crawler from near-zero to $200–400 a month once you need residential proxies. The larger cost is discovery risk: a cease-and-desist against a revenue-generating product forces an emergency migration to a paid API at exactly the moment you cannot afford the downtime. Budget the API price now instead of the incident later.

Can I scrape a site commercially if its robots.txt allows everything? Not necessarily. robots.txt only says which paths crawlers may fetch; the terms of service say what you may do with the data. A permissive robots.txt combined with a "personal, non-commercial use only" clause still puts a paid product in breach. Check both, and record the answer in version control so the constraint survives staff changes.

How do I scale throughput without breaking the crawl delay? Scale across hosts, never within one. Keep a per-host gate like the HostGate above and run many hosts concurrently; fifty hosts at a 10-second delay give you 18,000 pages an hour while each site sees one polite request every ten seconds. If you need more from a single host, that is the moment to buy the API rather than shard your IP space.

What happens to my cached data if I lose API access? That depends entirely on the caching clause. Terms that permit a bounded cache usually also require deletion on termination, so design your store with a source tag and a TTL from day one and keep the delete path tested. Never build a pricing tier whose value comes from data you are contractually obliged to erase when the vendor relationship ends.

Is a paid API cheaper than scraping at low volume? Almost always, once you cost your own time. A $29–99 monthly plan replaces a week of build plus roughly two hours a month of selector maintenance; at any freelance rate above $30 an hour the API is cheaper in month one and the gap widens forever. Scraping only wins on raw cash when the data has no commercial API at any price.

Same track:

Other tracks: