Web Scraping Guide: How to Choose the Right Approach

Almost every web scraping guide answers the same question: how do I parse this page? That question has been answered thoroughly, in every language, for fifteen years.

The question that actually costs teams money is different: which approach does this project need, and what will break when it grows?

Getting that wrong is expensive in both directions. Reach for a headless browser to scrape a static documentation site, and you have bought a 1.5 GB dependency and 40× the CPU cost for nothing. Reach for requests to scrape a marketplace, and you will spend three weeks discovering TLS fingerprinting before you extract a single reliable row.

This guide maps the five approaches, what each one costs, and the exact failure that pushes you to the next tier. Code for each is included, but the code is not the point — the transition is.

Table of Contents

• First: is this legal?
• The five tiers at a glance
• Tier 0: Don't write a scraper yet
• Tier 1: requests + BeautifulSoup
• Tier 2: headless browsers
• Tier 3: fingerprints and proxies
• Tier 4: a managed scraping API
• Cost comparison
• Choosing, in three questions
• Troubleshooting reference
• Frequently Asked Questions
• Wrapping up
• Related reading

First: is this legal?

Short version: scraping publicly accessible data is broadly established practice, and US case law has generally been favourable on genuinely public data. That is not blanket permission. What matters in practice:

• Public vs. authenticated. Data behind a login is a different legal category. Circumventing authentication is where scraping stops being defensible.
• Personal data. Anything relating to an identifiable person triggers GDPR/CCPA obligations regardless of whether it was public.
• Terms of service. Often prohibits automated access. Contractual, not criminal — but it is a real business risk.
• Load. Hammering a server hard enough to degrade it moves you from "scraping" toward something you cannot defend.

We cover this properly in Is Web Scraping Legal?. Read it before a commercial project, not after.

The five tiers at a glance

Tier	Approach	Works when	Ends when
0	Manual / no-code	One-off, a few hundred rows	You need it again next week
1	requests + BeautifulSoup	Server-rendered HTML, no defences	The page renders client-side
2	Headless browser	JavaScript-rendered content	Anti-bot fingerprints your browser
3	Browser + fingerprint patching + proxies	Protected sites, moderate volume	Maintenance exceeds the value
4	Managed scraping API	Production pipelines at any scale	—

Most teams belong in tier 1 or tier 4. The middle tiers are where projects go to die slowly: technically working, permanently needing attention.

Tier 0: Don't write a scraper yet

If you need 200 rows once, a scraper is the wrong tool. A browser extension or a no-code extractor gets you the data in ten minutes with zero code to maintain. The instinct to automate is often more expensive than the manual task.

Choose this when: it is genuinely a one-off, the volume is small, and nobody will ask for it again.

It ends when: somebody asks for it again. That is the entire signal. The moment "can you refresh that?" appears, you need a repeatable pipeline, and every hour spent perfecting the manual approach is wasted.

We compare the tooling in How to Scrape Websites Without Writing a Single Line of Code.

Tier 1: requests + BeautifulSoup

The correct default for server-rendered HTML. Fast, cheap, and dependency-light.

import requests
from bs4 import BeautifulSoup

resp = requests.get(
    "https://example.com/products",
    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                           "AppleWebKit/537.36 (KHTML, like Gecko) "
                           "Chrome/126.0.0.0 Safari/537.36"},
    timeout=30,
)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "lxml")
for card in soup.select("div.product-card"):
    print({
        "name": card.select_one("h3").get_text(strip=True),
        "price": card.select_one(".price").get_text(strip=True),
    })

TIP: Diagnostic Shortcut: Disable JavaScript in your browser and load the target page. If the data is still present in the source HTML, you are in Tier 1. This one check saves more wasted engineering effort than any other diagnostic in web scraping.

How to know this tier fits: disable JavaScript in your browser and load the target page. If the data is still there, you are in tier 1.

It ends when you view the source and the content is missing. Not the rendered DOM — the actual HTML the server sent. If the data arrives via a fetch() call after page load, requests will never see it, and no amount of selector tuning changes that.

The intermediate step people miss: before escalating to a browser, open DevTools → Network → Fetch/XHR and look at what the page is calling. Sites that render client-side are usually calling their own JSON API, and that API is often unauthenticated. Hitting it directly is faster, cheaper and more stable than rendering the page. Always check for this before reaching for a browser.

Tier 2: headless browsers

When content genuinely requires JavaScript execution, you need a real browser.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(viewport={"width": 1920, "height": 1080})
    page.goto("https://example.com/products", wait_until="domcontentloaded")
    page.wait_for_selector("div.product-card", timeout=20_000)

    items = page.eval_on_selector_all(
        "div.product-card",
        """cards => cards.map(c => ({
            name: c.querySelector('h3')?.innerText,
            price: c.querySelector('.price')?.innerText
        }))""",
    )
    browser.close()

print(items)

Two things worth internalising here. Use wait_for_selector rather than sleep — fixed sleeps are simultaneously too slow on fast loads and too fast on slow ones. And extract inside the browser with eval_on_selector_all rather than pulling the full HTML back into Python to re-parse; it is meaningfully faster on large pages.

NOTE: Resource Overhead: Headless Chromium consumes 300–800 MB RAM per worker and 1.5 GB on disk. Compute overhead is 10–40× higher per page than simple HTTP requests.

It ends when you start getting blocked despite the page rendering correctly. Anti-bot systems fingerprint headless browsers on dozens of signals — navigator.webdriver, canvas rendering, plugin arrays, font enumeration. Playwright out of the box is trivially detectable.

More depth in Browser Automation for Web Scraping and Scraping Browser vs Python Requests.

Tier 3: fingerprints and proxies

This is where scraping becomes an ongoing engineering commitment rather than a task. You are now maintaining three layers simultaneously:

1. TLS Fingerprint — Matching OpenSSL/BoringSSL handshakes and JA3/JA4 hashes to real Chrome browsers using tools like curl_cffi:

   from curl_cffi import requests as cffi_requests
   
   resp = cffi_requests.get(url, impersonate="chrome126", timeout=30)
   

2. Browser Fingerprint — Managing 30+ automation patches to pass WebGL, Canvas, and WebRTC bot detection checks.

3. Residential Proxy Rotation — Routing requests through real user residential IP pools. Rather than building custom rotation logic, you configure residential proxy endpoints directly.

Here is how to configure MrScraper's Residential Proxy with sticky session support and geotargeting (verified against documentation):

import requests

# MrScraper Residential Proxy — doc-verified authentication & port 10000
sid = "session_0921"
proxy_url = f"http://YOUR_USERNAME-country-us-sessid-{sid}-sesstime-10:YOUR_PASSWORD@proxy.mrscraper.com:10000"

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

response = requests.get("https://example.com/products", proxies=proxies, timeout=30)
print(response.status_code)

The honest maintenance math: three to five engineer-weeks to build all of it, then permanent upkeep on someone else's release schedule. Chrome ships roughly every four weeks. Anti-bot vendors ship whenever they like.

The full breakdown is in How to Bypass Cloudflare When Web Scraping, which walks all four defensive layers with working code.

It ends when you notice your team is maintaining anti-detection infrastructure instead of building the product that consumes the data. For a scraping vendor, that infrastructure is the moat. For everyone else, it is overhead.

Tier 4: a managed scraping API

The tier-3 work does not disappear when you outsource it — it moves server-side, along with the maintenance. What changes for you is that extraction becomes a description of the data instead of an evasion stack plus a selector tree.

With MrScraper's Web Scraper API and official Python SDK:

pip install mrscraper-sdk

import asyncio
import os

from mrscraper import MrScraper

client = MrScraper(token=os.getenv("MRSCRAPER_API_TOKEN"))

async def main():
    run = await client.create_scraper(
        url="https://example.com/products",
        message=(
            "Extract every product in JSON format: name, price (numeric), "
            "currency (ISO code), rating, in_stock (boolean), product_link."
        ),
        agent="listing",        # "general" | "listing" | "map"
        proxy_country="US",
    )

    if run["status"] != "Finished":       # Finished | Processing | Failed
        raise RuntimeError(run.get("error") or run["status"])

    print(run["tokenUsage"], run["runtime"])
    return run["scraperId"]

SCRAPER_ID = asyncio.run(main())

Note what is absent: no header set, no impersonate target to re-pin, no proxy rotation logic, no selectors. The agent parameter picks the behaviour — general parses one page, listing walks a result grid with pagination, map returns discovered URLs.

Reuse it across URLs without re-describing the schema, and submit at volume as one job rather than a loop:

job = await client.bulk_rerun_ai_scraper(
    scraper_id=SCRAPER_ID,
    urls=product_urls,
)

while job["status"] not in ("Finished", "Failed"):
    await asyncio.sleep(15)
    job = await client.get_result_by_id(job["id"])

rows   = job["data"]["mergedData"]      # flattened, one row per URL
failed = [d for d in job["data"]["urlDetails"] if d["status"] != "Finished"]

urlDetails is the part that matters in production: partial failure becomes a short retry list rather than a job you rerun from zero.

When this is the wrong choice: if you are learning, or scraping a handful of unprotected pages, tier 1 is genuinely better. A managed API earns its place when reliability has a cost — when someone downstream depends on the data arriving.

Cost comparison

Assume 100,000 pages per month from a moderately protected site:

Metric	Tier 1 (requests)	Tier 2 (Headless Browser)	Tier 3 (DIY Anti-Bot)	Tier 4 (Managed API)
Build time	Hours	1–2 days	3–5 weeks	Hours
Ongoing maintenance	Selector fixes	Selectors + browser infra	Continuous	None
Infrastructure	Negligible	Browser workers	Workers + proxy GB	Included
Success rate on protected sites	~0%	Low	Moderate–high	High
Breaks when	Page renders client-side	Anti-bot detects you	Chrome/vendor updates	—
Fails silently?	Yes — selectors return None	Yes	Yes	Status field is explicit

That last row is underrated. The most expensive scraping failures are not crashes — they are the runs that return a valid-looking 200 with wrong or stale data and write it into your database unnoticed. Whatever tier you pick, assert on the data, not the status code.

Choosing, in three questions

1. Does the data survive with JavaScript disabled? Yes → tier 1. Stop there.
2. Is there an unauthenticated JSON endpoint behind the page? Check DevTools → Network. Yes → call it directly, still tier 1.
3. Does anti-bot block you once volume rises? Yes → tier 3 or 4. Decide by asking whether unblocking is part of your product or part of your overhead.

Most projects that end up in tier 3 got there by skipping question 2.

Troubleshooting reference

Symptom	Likely tier problem	Fix
Selectors return None, HTML looks short	Client-rendered content	Check for a JSON endpoint, then escalate to tier 2
403 on the very first request	TLS fingerprint mismatch	curl_cffi with impersonate
Works locally, fails in production	Datacenter IP reputation	Residential exits, geo-matched
Passes 50 requests then blocks	Per-IP rate limiting	Backoff, wider pool, slower cadence
Browser works but gets challenged	Headless fingerprint detected	Patch automation tells, or move to tier 4
200 OK but fields are empty or stale	Decoy page served to bots	Assert on data; capture a screenshot
Scraper breaks after a site redesign	Selector coupling	Prompt-based extraction removes the coupling
Costs spiralled unexpectedly	Browser rendering everything	Audit which pages actually need JS

Frequently Asked Questions

What is web scraping?

Programmatically extracting data from websites — requesting pages, parsing the response, and turning it into structured records. It ranges from a 20-line script pulling a table to distributed infrastructure collecting millions of pages daily, and the engineering involved differs enormously across that range.

Which language is best for web scraping?

Python has the deepest ecosystem (BeautifulSoup, Scrapy, Playwright), which is why most tutorials use it. But the language is rarely the bottleneck — anti-bot defences and maintenance are. Node.js, Go, C# and PHP all have capable stacks; use what your team already runs in production.

Do I always need a headless browser?

No, and defaulting to one is a common and expensive mistake. Disable JavaScript and reload the page: if the data is still there, a simple HTTP client is 10–40× cheaper per page. Check for an underlying JSON API before escalating.

How do I scrape without getting blocked?

In order of impact: use residential IPs rather than datacenter ranges, match your TLS fingerprint to a real browser, keep request rates human, and respect robots.txt and rate limits. Volume triggers blocks far more often than any single request does. See How to Scrape Websites Without Getting Blocked.

When should I stop building my own scraper?

When maintenance stops being incidental. Concretely: you are re-pinning fingerprints on Chrome's schedule, you have a proxy pool to tune, and a person is on the hook when a site redesigns. At that point you are running scraping infrastructure — worth owning only if that is your product.

How much does web scraping cost?

Tier 1 is essentially free beyond your time. Tier 2 adds compute for browser workers. Tier 3 adds residential proxy bandwidth (billed per GB, and a JS-heavy page can pull 2–5 MB) plus continuous engineering. Tier 4 converts all of it into a predictable per-request cost. The expense people underestimate most consistently is maintenance, not infrastructure.

Wrapping up

The library is not the decision. The decision is which tier your project belongs to, and being honest about when it has moved:

• Test with JavaScript disabled first — it settles tier 1 vs. tier 2 in thirty seconds
• Look for the JSON endpoint before reaching for a browser
• Browsers are a cost decision, not a capability decision
• Tier 3 is a commitment, not a task — enter it deliberately
• Assert on the data, never on the status code, at every tier

Ready to skip the bypass engineering? Try MrScraper free — 1,000 free tokens, no credit card required. Scraping at volume? Book a call and we'll size it with you.

Related reading

• Is Web Scraping Legal? — The compliance groundwork to settle before a commercial project, whichever tier you land in.
• Python Web Scraping: The Complete Developer's Guide — The tier 1 and 2 toolchain in depth, with working Python throughout.
• How to Bypass Cloudflare When Web Scraping — Everything tier 3 involves, layer by layer, with the maintenance cost made explicit.
• How to Scrape Websites Without Getting Blocked — Rate limiting, pacing and detection avoidance that apply at every tier.
• Browser Automation for Web Scraping — Choosing between Playwright, Puppeteer and Selenium once you are in tier 2.
• Scraping Browser vs Python Requests — A direct cost and capability comparison of the tier 1 to tier 2 jump.
• Web Scraping API Guide — How managed APIs work under the hood and when they pay for themselves.
• Web Scraping API vs DIY Scraper — The build-versus-buy decision with the numbers laid out.
• MrScraper Documentation — Full API reference for the AI scraper, Python SDK, proxy endpoints and integrations.