Python Web Scraping in 2026: httpx + BeautifulSoup + Playwright — When to Use Which

Most "Python web scraping" tutorials from 2018 still recommend requests + BeautifulSoup. The requests part is fine but there are now better defaults, and half the modern web renders content in JavaScript so the classic stack silently returns empty pages.

This guide covers the 2026 decision tree: when httpx + BeautifulSoup is right (still 90%+ of sites), when to reach for Playwright, how to avoid getting blocked, and the two rules that keep your scraper both legal and ethical.

The decision tree in one image

That's it. Ninety percent of scraping tasks are static HTML — the classic stack (renamed) is the right tool. The rest need JS execution and Playwright is the modern answer.

Step 1: httpx + BeautifulSoup (the 90% case)

httpx is the drop-in replacement for requests. Same API, plus HTTP/2 support and native async. BeautifulSoup is unchanged from 2015 — parser choice matters (lxml is 5x faster than the default).

Key choices:

• `headers={"User-Agent": ...}` — always set a real UA (a contact address is polite). Default httpx UA is often blocked.
• `timeout=10` — never make an unbounded request; hung connections leak file descriptors.
• `raise_for_status()` — raises on 4xx/5xx so failures are loud, not silent.
• `"lxml"` parser — 5x faster than the default html.parser, and more forgiving with malformed HTML.
• `soup.select("css-selector")` — treat scraping as "pretend you're using JavaScript's document.querySelectorAll". CSS selectors are usually what you want, not the tree-walking API.

Step 2: Playwright (when JS renders the content)

Many modern SPAs (React, Vue, Angular) ship an empty <div id="root"> and hydrate everything client-side. httpx.get gets you an empty shell — the data you want is loaded via fetch after page load.

Test: httpx.get(url).text — search for the data you want. If it's not there, the site needs JS execution.

Install: pip install playwright && playwright install chromium (the browser download is ~150MB — worth it once, painful on every CI run without caching).

Cost: Playwright is 10-50x slower than httpx (page load + JS execution + browser overhead). Only reach for it when the data isn't in the static HTML.

The four ways to get blocked

Most anti-scraping systems watch for these four fingerprints:

1. Missing / obvious User-Agent

2. No delay between requests

Raw sequential requests hammer a server. Add a delay proportional to their response time — if a page takes 500ms to serve, wait 500-2000ms between requests. asyncio.sleep(random.uniform(0.5, 2.0)).

3. Requesting /robots.txt disallowed paths

Some sites use robots.txt for legit rate signals — check before scraping:

4. TLS / browser fingerprint mismatch

Advanced anti-scraping (Cloudflare, DataDome) fingerprint your TLS handshake and JS environment. httpx looks nothing like Chrome at the TLS layer. Options:

• curl_cffi — a curl-impersonate-based library that mimics real browser TLS.
• Playwright — real browser TLS by definition.
• Rotate residential proxies — the nuclear option.

For 99% of use cases (non-hostile sites, moderate volume, no login required), httpx + real UA + delay + robots.txt respect is enough. If you're scraping Cloudflare-protected sites at scale, you need real browsers or paid unblocking services.

Rate limiting yourself (the semaphore + sleep pattern)

For bulk scraping, combine asyncio.Semaphore (max concurrent) with asyncio.sleep (per-request delay):

Capping at 5-10 concurrent + 500ms-2s per-request delay keeps you well below what triggers rate-limit alarms on most sites.

Legal + ethical rules

1. Read the site's Terms of Service. Many prohibit scraping outright — you can still do it (ToS is contract law, not criminal law in most jurisdictions), but you take on legal risk. LinkedIn v. hiQ (2022) established that scraping publicly-visible data isn't a CFAA violation in the US, but wearing a hoodie doesn't make you invisible.

2. Don't scrape personal data. GDPR / CCPA / everything else penalises collecting personal information without consent. If you're building a dataset of "employees at Company X", stop.

Ethical additions everyone forgets:

• Send a real contact address in your UA. If your bot causes trouble, the site owner should be able to email you.
• If you cache the site's data, cache aggressively — don't hit their servers on every run.
• Prefer public APIs even if they're paid. Cheaper than legal risk.

The 2026 tooling summary

| Task | Reach for |

|---|---|

| Static HTML pages | httpx + BeautifulSoup(html, "lxml") |

| JS-rendered SPA | Playwright (headless Chromium) |

| Cloudflare / anti-bot | curl_cffi or Playwright + residential proxy |

| High-volume scraping | httpx.AsyncClient + Semaphore + sleep |

| Structured API data | httpx.get + .json() — no soup needed |

Web scraping in 2026 is 90% still httpx + BeautifulSoup, plus knowing when to escalate to a real browser. Master the polite-scraper habits (real UA, semaphore, sleep, robots.txt) and you'll build durable data pipelines without getting blocked or sued.

Next step: the Data Science track covers scraping, ETL, and pandas end-to-end — you'll build a real end-to-end pipeline from HTTP fetch to DataFrame to CSV export.