Managed Web Fetching with MrScraper: Rendering, IP Management, and Responsible Collection

Collecting public web data reliably involves more than sending an HTTP request. Pages are built with JavaScript, content changes by region, sessions rely on cookies, and a single network connection can fail or be slow. Managed web fetching services handle that infrastructure so teams can concentrate on the data.

This article explains what MrScraper's managed fetching layer does, where it helps, and how to use it responsibly. It does not replace the site's own rules. How much you collect, how often, and from whom remain your decisions, and they should stay within the limits of the sites and the law.

Note: This article describes infrastructure for collecting publicly available data. It is not a guide to getting around access controls on sites that do not permit automated collection.

What managed web fetching does

A managed fetching service sits between your code and the target site and takes care of the operational work of loading a page:

• IP routing: Requests are sent through a network of IP addresses, including residential IPs, so you do not have to run your own proxy infrastructure. IP rotation can be automatic.
• JavaScript rendering: Pages are loaded in a cloud-hosted browser so that content built by JavaScript is present in the result.
• Session and cookie handling: Cookies are managed across a browsing session, as a normal browser would.
• Geographic selection: You can choose the country a request appears to come from, which matters when a site shows different prices, currencies, or language by region.
• Structured delivery: Results can be returned as rendered HTML or as structured JSON or CSV, depending on the product you use. See the Web Scraper API and Web Rendered pages for current capabilities and the documentation for API details.

Why teams use it

• Less infrastructure to run: Proxy pools, headless browsers, and retries are time-consuming to build and maintain yourself.
• Complete page content: Rendering means prices, stock levels, and other JavaScript-loaded details appear in the result.
• Localized views of public content: Researchers can see what shoppers in a given country see, for example local prices and delivery options.
• Operational resilience: Routing across multiple IPs helps a job continue when an individual connection fails or is slow.

What IP rotation does and does not do

Rotating IP addresses changes where requests come from. It does not change how much load your job places on a site. A site that asks automated clients to slow down is asking about total request volume, and using more IP addresses does not make heavy collection any lighter for the site. Set your concurrency, delays, and schedule based on what the site can reasonably handle, not on how many IPs you have available.

Geographic selection and regional restrictions

Choosing a country is appropriate for seeing localized versions of content a site publishes openly, such as regional prices. It is not a way to reach content that a site or rights holder deliberately restricts to certain regions, for example for licensing or legal reasons. If a site limits content by region, respect that limit and look for an authorized route.

Collecting responsibly

Managed fetching makes collection easier to operate, so it is worth being deliberate about how you use it:

• Collect public data only: Limit collection to information any visitor can see without signing in, and avoid personal data about individuals.
• Review the site's rules: Read the terms of use and robots.txt, and follow any restrictions that apply to your use case.
• Prefer official sources: If the site offers a documented API, data feed, or licensing program, use it.
• Identify your traffic honestly: Where practical, use a descriptive user agent with a way to contact you, rather than presenting your traffic as something else.
• Keep volumes reasonable: Use delays, concurrency limits, and the lowest schedule frequency that meets your needs.
• Treat refusals as information: If a site responds with 403 or 429, slow down, honor any Retry-After header, and reconsider whether this collection is permitted.

Example: a polite fetch wrapper

The following Python example shows a request pattern that respects robots.txt, identifies itself, honors Retry-After, backs off on errors, and stops instead of retrying indefinitely. It works the same way whether you call a site directly or send the request through a managed service.

import time
import requests
from urllib import robotparser
from urllib.parse import urlparse

USER_AGENT = "example-research-bot/1.0 (contact: data-team@example.com)"
MAX_ATTEMPTS = 4

def allowed_by_robots(url: str) -> bool:
    parts = urlparse(url)
    rp = robotparser.RobotFileParser()
    rp.set_url(f"{parts.scheme}://{parts.netloc}/robots.txt")
    rp.read()
    return rp.can_fetch(USER_AGENT, url)

def polite_get(url: str) -> requests.Response | None:
    """Fetch a public page with limited retries and respect for server signals."""
    if not allowed_by_robots(url):
        print("Skipping: robots.txt does not allow this URL.")
        return None

    delay = 2.0
    for attempt in range(1, MAX_ATTEMPTS + 1):
        response = requests.get(url, headers={"User-Agent": USER_AGENT}, timeout=30)

        if response.status_code == 200:
            return response

        if response.status_code in (403, 429):
            retry_after = response.headers.get("Retry-After")
            wait = float(retry_after) if retry_after and retry_after.isdigit() else delay
            print(f"Attempt {attempt}: received {response.status_code}. Waiting {wait:.0f}s.")
            time.sleep(wait)
            delay *= 2
            continue

        response.raise_for_status()

    print("Stopping after repeated refusals. Review access, or look for an official API.")
    return None

The wrapper stops after a few attempts instead of looping. Repeated 403 or 429 responses are a reason to review whether and how you collect from that site, not a reason to escalate.

Plan, observe, adapt

A reliable collection workflow follows a simple pattern:

• Plan: Decide what you need, check the site's rules and any official data sources, and set limits for concurrency, delays, and frequency.
• Observe: Track failed requests, response codes, and changes in page content so you notice problems early.
• Adapt: Slow down, back off, or pause when a site signals that you should, and fix the cause instead of retrying harder. This turns an error from a failure to be overcome into a signal to respond to, which tends to make pipelines more dependable and keeps them on good terms with the sites they depend on.