Agentic vs Deterministic Web Data Workflows: How to Choose

Many teams now ask the same question: should an AI agent run our web data collection?

Sometimes the answer is yes. Often a plain, fixed pipeline does the job faster and for less money.

This guide runs the same task both ways. You will see where each approach fits, how to measure them side by side, and how to send unclear results to a person for review.

What is the difference?

A deterministic workflow follows the same steps every time. You give it a list of URLs and a schema. It collects the pages, extracts the fields and saves the records. The same input takes the same path.

An agentic workflow lets a model choose some of the steps. The model might decide which page to open, which link to follow, or when to stop. The path can change from one run to the next.

Neither approach wins in every case. The right choice depends on how much you know about your sources before the job starts.

The example task

We will use one task for both versions:

For each supplier on a list, collect the return window in days, any restocking fee, and the policy text that supports those values.

In version A, you already know the URL of each supplier's return policy page.

In version B, you only know each supplier's homepage.

That one difference changes which workflow makes sense.

When is a fixed extraction pipeline enough?

A fixed pipeline is enough when you can answer "yes" to most of these questions:

• Do you know the exact pages to collect?
• Do the pages hold the same kind of content?
• Can you describe the fields you need in a schema?
• Can code check whether a record looks right? Version A passes all four. You have the URLs. Every page is a return policy. The fields are clear. Code can check that the return window is a number.

Build the fixed pipeline

Here is a simple version in Python. It sends each URL to the MrScraper Scrape JSON endpoint with a prompt and a JSON schema. MrScraper fetches the page and returns structured data. Your code checks the result and stores it.

import os
import requests
 
MRSCRAPER_URL = "https://api.mrscraper.com/"
TOKEN = os.environ["MRSCRAPER_API_TOKEN"]
 
POLICY_PROMPT = """Extract the return policy details from this page.
Json Schema:
{
  "type": "object",
  "properties": {
    "return_window_days": {"type": ["integer", "null"]},
    "restocking_fee": {"type": ["string", "null"]},
    "policy_text": {"type": ["string", "null"]}
  }
}"""
 
 
def extract_policy(url: str) -> dict:
    response = requests.post(
        MRSCRAPER_URL,
        params={"token": TOKEN},
        headers={"x-api-token": TOKEN, "Content-Type": "application/json"},
        json={
            "url": url,
            "prompt": POLICY_PROMPT,
            "agent": "general",
            "maxRetries": 2,
        },
        timeout=330,
    )
    response.raise_for_status()
    return response.json()

The response holds more than the extracted data. It also includes fields such as error, runtime and token_usage. Keep the ones you need for checks and cost tracking.

Hand the result to your application

Now turn each response into a record your app can store. Add the source URL and the time you collected it. You will need both later for review and debugging.

from datetime import datetime, timezone
 
 
def to_record(supplier_id: str, url: str, result: dict) -> dict:
    data = result.get("data") or {}
    return {
        "supplier_id": supplier_id,
        "source_url": url,
        "retrieved_at": datetime.now(timezone.utc).isoformat(),
        "return_window_days": data.get("return_window_days"),
        "restocking_fee": data.get("restocking_fee"),
        "policy_text": data.get("policy_text"),
        "api_error": result.get("error"),
        "runtime": result.get("runtime"),
        "token_usage": result.get("token_usage"),
    }
 
 
records = [
    to_record(s["id"], s["policy_url"], extract_policy(s["policy_url"]))
    for s in suppliers
]

Why this works well

• Each URL gets one call, so cost is easy to predict.
• Every run follows the same path, so bugs are easy to repeat and fix.
• Failures are plain: a page did not load, or a field is missing. If your task looks like version A, start here. An agent would add cost and new ways to fail without adding much value.

When does an agent add value?

An agent helps when the job needs choices you cannot write down ahead of time. Look for these signs:

• You know the site, but not the exact page.
• Sites use different layouts and link names.
• The answer may sit across two or three pages.
• Someone has to decide when enough information has been found. Version B fits this pattern. One supplier links to "Returns" in the footer. Another puts it under "Help," then "Shipping and Returns." A fixed list of URLs cannot cover that.

Build a bounded agent

"Bounded" means the agent has firm limits. It can only use the tools you give it. It has a page budget. It must stop and report back when the budget runs out.

In this sketch, the agent has two tools. Both call MrScraper:

• read_page(url) returns a page as Markdown through the Scrape Markdown endpoint. The model reads it to find links and text.
• extract_policy(url) is the same function from version A. The agent calls it once it finds the right page. The choose_next_step function is your own model call. MrScraper does not make this choice. Your agent framework or LLM does.

from urllib.parse import urlparse
 
PAGE_BUDGET = 5
 
 
def read_page(url: str) -> str:
    response = requests.post(
        MRSCRAPER_URL,
        params={"token": TOKEN, "markdown": "true"},
        headers={"x-api-token": TOKEN, "Content-Type": "application/json"},
        json={"url": url, "maxRetries": 2},
        timeout=330,
    )
    response.raise_for_status()
    return response.json().get("markdown", "")
 
 
def same_site(url: str, homepage: str) -> bool:
    return urlparse(url).netloc == urlparse(homepage).netloc
 
 
def find_policy(supplier_id: str, homepage: str) -> dict:
    visited = []
    url = homepage
 
    for _ in range(PAGE_BUDGET):
        page_text = read_page(url)
        visited.append(url)
 
        # Your model call. It returns one of:
        # {"action": "extract", "url": "..."}
        # {"action": "follow", "url": "..."}
        # {"action": "give_up"}
        step = choose_next_step(
            goal="Find the supplier's return policy page.",
            current_url=url,
            page_text=page_text,
            visited=visited,
        )
 
        if step["action"] == "give_up":
            break
        if not same_site(step["url"], homepage):
            break
        if step["action"] == "extract":
            record = to_record(supplier_id, step["url"], extract_policy(step["url"]))
            record["path"] = visited
            return record
 
        url = step["url"]
 
    return {"supplier_id": supplier_id, "status": "not_found", "path": visited}

Look at what stays fixed, even in the agent version. The schema, the record format and the site boundary are all set by code. Only the page search is up to the model.

This is a good pattern. Let the model handle the uncertain part, and keep code in charge of everything else.

How do you compare latency, cost and failure modes?

Do not choose based on a hunch. Run both versions on the same sample and compare the numbers.

Set up a fair test

1. Pick 30 to 50 suppliers.
2. Find the correct answer for each one by hand. This is your ground truth.
3. Run version A with the known policy URLs.
4. Run version B with only the homepages.
5. Log every call: the URL, the runtime, the token usage and the final record.

What to measure

Treat wrong fields and missing fields as two separate problems. A missing value is easy to catch. A wrong value that looks right is much harder to spot. An agent that extracts from the wrong page can produce exactly this kind of error.

Common failure modes

Fixed pipelines tend to fail in plain ways. The page moved, the layout changed, or a field disappeared. You see it in the error field or in your checks.

Agents can fail in quieter ways:

• They follow a link to the wrong page and extract from it.
• They use the whole page budget and find nothing.
• They stop too early because a page looked close enough.
• They take different paths on the same input. Log the full path the agent took. You will need it when you debug.

A hybrid is often the best answer

Many teams end up using both. The agent runs once to find the right page for each supplier. The app saves those URLs. From then on, the fixed pipeline collects the data.

The agent only runs again when the fixed pipeline fails for a supplier. This keeps daily cost low and still handles new or changed sites.

If you run the fixed pipeline often, MrScraper's Schedule feature can run saved scrapers on a timetable. Check the docs to see whether it fits your setup.

How do you route uncertain cases to review?

No workflow gets every record right. Plan for a review step from day one.

Decide what "uncertain" means

Write rules that code can check. For this task, flag a record when:

• The API returned an error.
• The return window is missing or outside 0 to 365 days.
• The policy text is missing.
• The return window number does not appear in the policy text.
• The agent used its full page budget without finding a page.
• Version A and version B disagree for the same supplier.

Send flagged records to a queue

def review_reasons(record: dict) -> list[str]:
    reasons = []
    if record.get("status") == "not_found":
        reasons.append("policy page not found within budget")
    if record.get("api_error"):
        reasons.append("API returned an error")
 
    days = record.get("return_window_days")
    text = record.get("policy_text") or ""
 
    if not text:
        reasons.append("missing policy text")
    if days is None:
        reasons.append("missing return window")
    elif not 0 <= days <= 365:
        reasons.append("return window out of range")
    elif str(days) not in text:
        reasons.append("return window not found in policy text")
 
    return reasons
 
 
for record in records:
    reasons = review_reasons(record)
    if reasons:
        review_queue.append({**record, "reasons": reasons})
    else:
        save_record(record)

Some flags will be false alarms. A policy might say "thirty days" instead of "30 days." That is fine. A person can clear it in seconds.

The reviewer opens the source URL, checks the value, and approves or fixes it. Save each fix as new ground truth. Your test set grows over time, and you can measure whether each change helps.

A quick decision guide

FAQs

When is a fixed extraction pipeline enough?

A fixed pipeline is enough when you know which pages to collect, the pages hold similar content, and code can check each record. Most repeat data collection jobs fit this pattern.

How do you route uncertain cases to review?

Write clear rules for what counts as uncertain. Examples include missing fields, values outside a normal range, and values that do not appear in the source text. Send matching records to a queue with the URL and the extracted data so a person can check them.

Which parts does MrScraper provide, and which parts must my application implement?

MrScraper is a developer-focused web data API. It fetches pages, can capture content that loads in the browser, returns Markdown or HTML, and extracts structured JSON from your prompt and schema. It also offers stored results, scheduling, webhooks, SDKs and an MCP server.

Your application owns the rest. That includes the agent and its model calls, the page budget and stop rules, validation, the review queue and your database. Your team also decides which sources it has permission to collect from.