Scraping JavaScript-Rendered Sites with Playwright and Python

Some sites render their data with JavaScript after the page loads, so a plain HTTP request returns an empty shell and your parser finds nothing. When there is no official API and the data lives behind client-side rendering, a headless browser is the tool that actually sees what a user sees. This page shows a runnable async Playwright scraper, how to harden it for production, how to stay polite under rate limits, what it costs to run at volume, and — most importantly — when scraping is justified at all. It is part of the Web Scraping vs Official APIs guide, which makes the broader case that official endpoints almost always win.

Treat a headless browser as the most expensive tool in your automation kit. It is the right call surprisingly rarely, and knowing when to skip it is worth more than any selector trick.

When Playwright is the right tool (and when it is not)

Playwright drives a real Chromium, Firefox, or WebKit browser. It executes JavaScript, waits for network calls to settle, and exposes the fully rendered DOM. That power costs you memory, CPU, and fragility, so reach for it only when lighter options fail.

The order matters. Before launching a browser, open the page's network tab and look for the XHR/fetch call that actually returns the data — it is frequently a clean JSON endpoint you can hit directly, which is faster, cheaper, and far less brittle than scraping rendered HTML. Because that endpoint returns structured data, you skip DOM parsing entirely and go straight to validating the payload, ideally with Pydantic v2. The decision below is the one to run through every time before you pip install playwright.

Every step up this tree multiplies your running cost by roughly an order of magnitude, so exhaust the cheaper branches honestly before landing on the browser.

The scraping pipeline

A headless browser scrape moves through a fixed sequence: launch the browser, route traffic through a proxy if you have one, navigate to the target, wait for the meaningful selector to appear, extract the data, and close cleanly. Most failures happen at the wait step — navigate too eagerly and you read an empty shell; wait on the wrong signal and you time out on pages that are actually fine. The diagram below shows the path each run follows.

A runnable async Playwright scraper

Install the package and a browser binary first: pip install playwright then playwright install chromium. Everything below is driven by environment variables, so the target and proxy are never hardcoded. Because the API is async-native, it composes cleanly with the rest of an async httpx toolchain.

import os
import asyncio
import logging

from playwright.async_api import async_playwright, TimeoutError as PWTimeout

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
logger = logging.getLogger("scraper")

TARGET_URL = os.environ["SCRAPE_TARGET_URL"]          # the JS-rendered page
WAIT_SELECTOR = os.getenv("SCRAPE_WAIT_SELECTOR", ".item")
PROXY_SERVER = os.getenv("SCRAPE_PROXY_SERVER")        # e.g. http://gateway:8000 or None
PROXY_USER = os.getenv("SCRAPE_PROXY_USER")
PROXY_PASS = os.getenv("SCRAPE_PROXY_PASS")
NAV_TIMEOUT_MS = int(os.getenv("SCRAPE_NAV_TIMEOUT_MS", "20000"))
POLITE_DELAY_S = float(os.getenv("SCRAPE_POLITE_DELAY_S", "2.0"))


def _proxy_config() -> dict | None:
    if not PROXY_SERVER:
        return None
    cfg = {"server": PROXY_SERVER}
    if PROXY_USER and PROXY_PASS:
        cfg.update(username=PROXY_USER, password=PROXY_PASS)
    return cfg


async def scrape() -> list[dict]:
    async with async_playwright() as pw:
        browser = await pw.chromium.launch(headless=True, proxy=_proxy_config())
        context = await browser.new_context(
            user_agent=os.getenv("SCRAPE_USER_AGENT", "Mozilla/5.0 (compatible; builder-bot/1.0)"),
        )
        page = await context.new_page()
        try:
            await page.goto(TARGET_URL, wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS)
            # Wait for the JS-rendered content, not just the page shell.
            await page.wait_for_selector(WAIT_SELECTOR, timeout=NAV_TIMEOUT_MS)

            items = await page.query_selector_all(WAIT_SELECTOR)
            results = []
            for item in items:
                title = await item.inner_text()
                results.append({"title": title.strip()})

            logger.info("Extracted %d records", len(results))
            return results
        except PWTimeout:
            logger.error("Selector %r never appeared — layout changed or blocked", WAIT_SELECTOR)
            return []
        finally:
            await context.close()
            await browser.close()
            # Politeness: pace requests so you do not hammer the target.
            await asyncio.sleep(POLITE_DELAY_S)


if __name__ == "__main__":
    rows = asyncio.run(scrape())
    print(rows)

The two details that separate a working scraper from a flaky one are wait_for_selector and the proxy block. Waiting on the content selector — not a fixed sleep, not networkidle alone — is what handles the JavaScript render reliably. A networkidle wait looks tempting, but on pages with long-polling, analytics beacons, or streamed responses the network never goes idle and you time out on a page that rendered fine seconds earlier. Waiting on the selector that holds your data sidesteps all of that. The proxy config is read from the environment and silently skipped when absent, so the same code runs locally without a proxy and in production with one.

Hardening the scraper for production

The snippet above works on your laptop. Three additions make it survive an unattended nightly run. First, block the resources you do not need. A scraper almost never cares about images, fonts, media, or third-party ad and analytics scripts, and those are the bulk of the bytes and CPU a page burns. Route them to abort() and each page loads faster on less memory. Second, retry the whole run on transient failures rather than losing the batch — the same exponential-backoff discipline covered in retrying failed HTTP requests with tenacity applies to a browser navigation. Third, cap concurrency: one browser with a handful of pages, not a page per URL, or you will exhaust RAM.

BLOCKED_TYPES = set(os.getenv("SCRAPE_BLOCK_TYPES", "image,font,media").split(","))


async def _block_heavy(route):
    if route.request.resource_type in BLOCKED_TYPES:
        await route.abort()
    else:
        await route.continue_()


async def fetch_page(context, url: str, selector: str) -> list[dict]:
    page = await context.new_page()
    await page.route("**/*", _block_heavy)
    try:
        await page.goto(url, wait_until="domcontentloaded", timeout=NAV_TIMEOUT_MS)
        await page.wait_for_selector(selector, timeout=NAV_TIMEOUT_MS)
        nodes = await page.query_selector_all(selector)
        return [{"title": (await n.inner_text()).strip()} for n in nodes]
    finally:
        await page.close()

The bandwidth saving is not marginal. Blocking images, fonts, and ad scripts on a typical content-heavy page cuts the transfer from roughly 2.8 MB to about 1.2 MB — more than half — and the CPU saving from not decoding images and running ad scripts is larger still. The chart below shows where the bytes go on a default load versus a hardened one.

One caution: block resources by type, not by wildcard-everything, and never block scripts on a JS-rendered page — the whole reason you launched a browser is to run that JavaScript. If your target's content depends on a specific stylesheet for layout that your selectors key off, keep CSS too. When these scrapes run on a schedule, wire them into your cron-driven data pipeline so retries, logging, and alerting are handled in one place rather than reinvented per script.

Staying polite and avoiding bans

A scraper that fires as fast as the event loop allows will get rate-limited, IP-banned, or served a CAPTCHA within minutes, and it is also simply rude. Politeness is both an ethics and a survival strategy, and the boundaries are laid out in full in respecting robots.txt and API terms of service.

• Respect robots.txt and the Terms of Service. Check what the site permits before you scrape it. Public, non-authenticated data is the safe zone; anything behind a login wall is not.
• Throttle deliberately. The POLITE_DELAY_S pause above paces requests. For multi-page crawls, add randomized jitter so your timing does not look robotic, and cap concurrency with a semaphore.
• Set a real, identifiable user agent. A custom UA that names your bot is more honest and often less likely to be blocked than a spoofed browser string.
• Cache aggressively. Store every successful response so a re-run does not re-fetch. This is the single biggest reduction in load on the target, and it mirrors the caching discipline from Best Practices for API Rate Limiting.
• Back off on 429 and 5xx. When the target pushes back, slow down with exponential backoff rather than retrying immediately — the same behaviour you would build for 429 Too Many Requests errors against any API.
• Use proxies sparingly and legally. Residential proxy rotation exists to distribute load, not to defeat anti-abuse systems. If you need to defeat them, that is a strong signal the site does not want to be scraped — reconsider.

The unglamorous truth: heavy scraping infrastructure (proxy pools, CAPTCHA solvers, browser farms) costs real money and real maintenance, which is exactly the total-cost-of-ownership argument the parent guide makes for preferring an official API whenever one exists.

Cost and performance at scale

The commercial case against scraping is a spreadsheet, not a principle. A hidden JSON endpoint answered by httpx returns in roughly 0.15 seconds on about 30 MB of process memory. Parsing static HTML with a lightweight parser lands near 0.4 seconds and 45 MB. A Playwright page — even hardened, with assets blocked — runs closer to 2.5 seconds and 380 MB, because you are paying for a full browser process, a rendering engine, and a JavaScript VM per context. That is the gap you sign up for.

Put dollars on it. Scraping 100,000 pages a month with Playwright at 2.5 seconds each is about 70 hours of single-threaded browser work; on a modest always-on box you parallelise that down to a night, but the memory ceiling caps how many contexts fit — a 2 GB instance holds maybe four or five browser pages at once, so you are renting a bigger machine or more of them. Add residential-proxy bandwidth at a few dollars per gigabyte and the monthly bill climbs into real money. The same 100,000 records pulled from a JSON endpoint fit comfortably on the smallest instance and cost cents. When the target has an API and you are still scraping to dodge a subscription fee, run that comparison honestly — the browser tax usually exceeds the API you were avoiding. If you are weighing building this pipeline against a hosted automation tool, Zapier vs Make vs Python works through the same build-versus-buy math.

Scraping versus an official API

When a vendor offers an API, use it. An API gives you structured JSON, a documented contract, versioning, a rate-limit policy you can plan around, and legal clarity. A scraper gives you fragile DOM parsing that breaks silently the next time the site ships a redesign. The reliability gap is enormous, and it widens at scale — a selector that shifts one class name takes your whole pipeline down at 3 a.m. with no error the vendor will ever help you debug.

Use Playwright scraping only as a bridge: a site you genuinely need has no API, the data is JS-rendered, and there is no hidden JSON endpoint to call. Even then, design the scraper behind an adapter interface so you can swap in an official API later without rewriting downstream logic — the adapter pattern is shown in the parent Web Scraping vs Official APIs guide. That single seam is what turns "the vendor finally shipped an API" from a rewrite into a one-file change.

Builder verdict

Prefer an official API every single time one is available — it is more reliable, cheaper to maintain, and legally cleaner. Before reaching for Playwright, hunt for the hidden JSON endpoint that feeds the page; you will find one more often than you expect, and httpx against it beats a browser on every axis by roughly ten to one on both time and memory. Keep Playwright in your toolkit strictly as the fallback for genuinely JavaScript-rendered sites with no API and no reachable endpoint. When you do scrape, block the heavy assets, scrape politely, cache everything, and hide it behind an adapter so the day an API appears, swapping to it is trivial.

FAQ

When should I use Playwright instead of httpx and BeautifulSoup? Only when the data is rendered by JavaScript after the page loads, or when you need to click, log in, or scroll to reveal it. If the data is in the raw HTML source, or a hidden JSON endpoint feeds the page, httpx is faster, cheaper, and far less fragile. A headless browser is the heavyweight fallback, not the default.

How much does it cost to scrape 100,000 pages a month with Playwright? Budget for compute plus proxy bandwidth, not just compute. At roughly 2.5 seconds and 380 MB per page you need a machine with real RAM to run a few contexts in parallel, plus residential-proxy bandwidth at a few dollars per gigabyte. The same volume pulled from a JSON endpoint costs cents on the smallest instance — often less than the API subscription you were trying to avoid.

How do I keep a Playwright scraper from getting banned? Throttle requests with deliberate delays and jitter, cap concurrency, set an honest user agent, respect robots.txt, back off on 429 and 5xx responses, and cache successful results so you never re-fetch. If you find yourself needing to defeat CAPTCHAs and rotate residential proxies, treat that as a signal the site does not want to be scraped.

Is scraping JS-rendered sites legal? Scraping publicly accessible, non-authenticated data is generally permissible if you respect the site's Terms of Service and robots.txt and comply with privacy law such as GDPR and CCPA. Bypassing authentication or anti-abuse measures moves you into risky territory. Official APIs always carry clearer legal footing.

How do I make the scraper cheaper to run without breaking it? Block images, fonts, media, and ad scripts through page.route, which typically halves bytes transferred and cuts CPU further, but never block the JavaScript the page needs to render. Cache every successful fetch, run on a schedule instead of continuously, and match your concurrency to available RAM so you are not renting a bigger box than the workload needs.

Related

Same track:

• Web Scraping vs Official APIs — the parent guide and the adapter pattern for swapping in an API later.
• Respecting robots.txt and API Terms of Service — where the legal and politeness lines actually sit.
• Scheduling Data Pipelines with Cron — run these scrapes unattended with retries and alerting.
• Zapier vs Make vs Python — build-versus-buy for the automation around the scrape.

Other tracks:

• httpx vs requests for async — the async HTTP client to reach for when there is no browser needed.
• Best Practices for API Rate Limiting — the throttling discipline that keeps you unbanned.