Headless Browser

A headless browser is a web browser that runs without a visible window. It loads pages, runs JavaScript and builds the page's DOM like a normal browser, but code controls it through tools such as Puppeteer, Playwright or Selenium.

Teams use headless browsers for automated testing, screenshots and PDFs, performance monitoring, and collecting content that only appears after JavaScript runs. Many pages don't need one, so check first whether a plain request will do.

How a headless browser renders a page

1. Fetch the HTML. The browser requests the page, just as a plain HTTP client would.
2. Build the DOM. It parses the HTML into a tree of elements.
3. Load resources. It fetches the CSS, JavaScript, images and fonts the page references.
4. Run scripts. JavaScript requests data, adds elements and changes the DOM, often well after the first HTML arrives.
5. Read the result. What you extract is the DOM at the moment you read it, so timing decides what you get.

Waiting for content

A page can look "loaded" before the data you need has appeared. Browser tools offer several wait conditions:

Pages with live chat, analytics or polling may never go fully idle, so always set a timeout. Waiting for the exact element you need is faster and more dependable than waiting for the network to settle.

Memory and cost

Each page runs a full rendering engine with its own JavaScript memory, so resource use grows quickly with concurrency and page weight.

• Limit concurrency. Run as many pages in parallel as your memory allows, not as many as your job has URLs.
• Close pages when you're done. Long-lived tabs keep growing as scripts run.
• Reuse the browser, isolate the context. Starting a browser is slow; opening a fresh context or page within one is cheaper.
• Skip what you don't need. Images, fonts and media often add load time without adding data.
• Set timeouts. A page that never finishes ties up memory and slows everything behind it.

Headless browser vs plain HTTP request

Alternatives to try first

• Check the page source. If the data is in the initial HTML, a plain request and an HTML parser are enough.
• Look for embedded structured data. Many pages include schema.org JSON-LD or other data in the HTML they send, such as product names, prices and ratings.
• Use an official API, feed or sitemap. These are built for outside tools and change less often than page layouts.
• Use the site's server-rendered pages. Some sites serve full HTML for categories, archives or print views. Calling a site's internal endpoints directly is a different matter, with its own stability and permission questions; see API reverse engineering.

How sites recognize automated browsers

Many sites check whether a visitor is a person or automation, for security (credential stuffing, fraud, account abuse), to protect capacity, and for compatibility with test tools. The W3C WebDriver specification defines navigator.webdriver, which a browser sets to true when automation controls it, so cooperating browsers can tell a page they're automated. Sites combine this with browser checks, network signals such as IP reputation, and request behavior.

Being recognized is expected behavior, not a defect to fix. If a site restricts automated access, the dependable route is permission: an official API, a data agreement or an allowlist arranged with the site owner.

When a headless request is denied

Headless rendering in MrScraper

• Web Fetching API: browserRendering=true renders the page in a headless browser; waitForSelector waits for a specific element; timeout caps the wait (default 30 seconds).
• Scraper API: browserRendering=true renders in Chromium; waitUntil accepts domcontentloaded (default), load, networkidle2 or networkidle0; screenshot=true returns an image of what the browser saw.

curl -G "https://api.mrscraper.com/" \
  -H "x-api-token: YOUR_API_TOKEN" \
  --data-urlencode "token=YOUR_API_TOKEN" \
  --data-urlencode "url=https://example.com/products" \
  --data-urlencode "browserRendering=true" \
  --data-urlencode "waitForSelector=.product-card" \
  --data-urlencode "timeout=60"

Web Fetching responses report the target site's status code in an x-status-code header, so you can spot denied requests and pause a job. Use it for publicly available pages, within each site's terms and MrScraper's Acceptable Use Policy.

Frequently asked questions

Do I need a headless browser for every data collection job?

No. If the data is in the page's initial HTML, embedded structured data, an API or a feed, a plain request is faster and cheaper.

Why does my headless browser return an empty or partial page?

Usually the content hadn't rendered when you read the page. Wait for the specific element that holds your data, and raise the timeout for slow pages.

How much memory does a headless browser use?

It depends on the page and how many run at once. Heavy pages and high concurrency use the most, so cap parallel pages and close each one when you're done.

Can websites detect headless browsers?

Often, yes. Standard flags such as navigator.webdriver exist so automated browsers can identify themselves. Plan around permitted access rather than concealment.

What should I do if a site blocks my headless browser?

Stop sending requests, check the status code and any reference ID, and review the site's terms. If you need the data, contact the site owner or use an official API.