Design Reliable Web Data Tools for AI Agents

An AI agent picks a tool by reading the tool's name, its description and its input schema. That is all it has to go on.

If a tool is vague, the agent guesses. It might call the wrong tool, pass a bad URL or get back a wall of HTML it cannot use. When the tool fails, the agent might not even notice.

Good tool design fixes most of this before you write a single prompt. In this guide, you will design three web data tools with clear contracts and clear failure states:

Tool	Job	MrScraper call behind it
`discover_pages`	Find pages on a known site	Scrape Sitemap with the `map` agent
`extract_page`	Pull named fields from one page	Scrape JSON with a prompt and schema
`get_stored_result`	Read more of an earlier result	Your own result store

The code is in Python, but the ideas work in any language or agent framework.

How should tool inputs be defined?

Start with the inputs. They tell the agent what the tool needs and stop bad calls before they happen.

Follow five rules

Give each tool one job. A tool called web that searches, extracts and summarizes is hard for a model to use well. Three small tools are easier.
Name tools with a verb. discover_pages tells the model what happens. site_tool does not.
Say when to use the tool and when not to. Put both in the description.
Use types, limits and required fields. A field with "minimum": 1, "maximum": 20 is clearer than a sentence that says "keep it small."
Keep settings in code. The model should not choose timeouts, retry counts or credentials. Your code sets those.

Write the tool definitions

These definitions use JSON Schema. Most function-calling APIs and MCP clients accept this format, though the wrapper around it differs by provider.

json

[
  {
    "name": "discover_pages",
    "description": "List page URLs on one website, starting from a known URL. Use this when you know the site but not the exact page. Do not use it to read page content.",
    "input_schema": {
      "type": "object",
      "properties": {
        "start_url": {"type": "string", "format": "uri", "description": "Full URL with https://"},
        "include_patterns": {"type": "array", "items": {"type": "string"}, "description": "Only keep URLs matching these regex patterns"},
        "max_urls": {"type": "integer", "minimum": 1, "maximum": 200, "default": 50}
      },
      "required": ["start_url"]
    }
  },
  {
    "name": "extract_page",
    "description": "Extract named fields from one page as JSON. Use this when you know the exact page URL and the fields you need. Do not use it to find pages.",
    "input_schema": {
      "type": "object",
      "properties": {
        "url": {"type": "string", "format": "uri"},
        "fields": {
          "type": "array",
          "minItems": 1,
          "maxItems": 20,
          "items": {
            "type": "object",
            "properties": {
              "name": {"type": "string", "pattern": "^[a-z][a-z0-9_]*$"},
              "type": {"type": "string", "enum": ["string", "number", "integer", "boolean"]},
              "description": {"type": "string"}
            },
            "required": ["name", "type", "description"]
          }
        }
      },
      "required": ["url", "fields"]
    }
  },
  {
    "name": "get_stored_result",
    "description": "Read the full stored output of an earlier extract_page call, one chunk at a time. Use this when a result says it was truncated or you need the page text as evidence.",
    "input_schema": {
      "type": "object",
      "properties": {
        "result_id": {"type": "string"},
        "offset": {"type": "integer", "minimum": 0, "default": 0}
      },
      "required": ["result_id"]
    }
  }
]

Check inputs in code too

Do not trust the model to follow the schema every time. Check each call before it reaches the API. This example uses Pydantic.

python

from pydantic import BaseModel, Field, HttpUrl
from typing import Literal
 
 
class DiscoverInput(BaseModel):
    start_url: HttpUrl
    include_patterns: list[str] = []
    max_urls: int = Field(50, ge=1, le=200)
 
 
class FieldSpec(BaseModel):
    name: str = Field(pattern=r"^[a-z][a-z0-9_]*$")
    type: Literal["string", "number", "integer", "boolean"]
    description: str
 
 
class ExtractInput(BaseModel):
    url: HttpUrl
    fields: list[FieldSpec] = Field(min_length=1, max_length=20)
 
 
class StoredResultInput(BaseModel):
    result_id: str
    offset: int = Field(0, ge=0)

If validation fails, return a clear error to the agent. You will see how in the errors section below.

What belongs in structured tool results?

The agent reads every tool result. A messy result wastes tokens and makes the next step harder.

Use one result shape for every tool

Give all three tools the same envelope. The agent learns one shape, and your code can log and check every call the same way.

json

{
  "ok": true,
  "data": {},
  "source": {"url": "https://example.com/pricing", "retrieved_at": "2026-10-15T06:30:00+00:00"},
  "usage": {"token_usage": 12, "runtime": 4.2},
  "truncated": false,
  "result_id": "r_7f3c9a",
  "error": null
}

Each part has a job:

ok tells the agent at a glance whether the call worked.
data holds only what the agent asked for.
source keeps the URL and time, so later steps can cite them.
usage passes through MrScraper's token_usage and runtime, so your app can track cost.
truncated and result_id tell the agent there is more, and how to get it.
error is null on success. On failure, it explains what went wrong.

Keep large content out of the result

A full page can be thousands of tokens. Most steps only need a few fields. So the extraction tool stores the full response in your app and returns a short result plus a result_id. The agent can call get_stored_result if it needs more.

Build the tools

First, a small helper for MrScraper calls. All three Scraper API endpoints share the same host and auth.

python

import os
import uuid
import requests
from datetime import datetime, timezone
 
MRSCRAPER_URL = "https://api.mrscraper.com/"
TOKEN = os.environ["MRSCRAPER_API_TOKEN"]
RESULT_STORE = {}  # Use a database in production.
 
 
def call_mrscraper(body: dict, **params) -> dict:
    response = requests.post(
        MRSCRAPER_URL,
        params={"token": TOKEN, **params},
        headers={"x-api-token": TOKEN, "Content-Type": "application/json"},
        json=body,
        timeout=330,
    )
    response.raise_for_status()
    return response.json()
 
 
def now() -> str:
    return datetime.now(timezone.utc).isoformat()
 
 
def envelope(data, url=None, raw=None, truncated=False, result_id=None):
    raw = raw or {}
    return {
        "ok": True,
        "data": data,
        "source": {"url": url, "retrieved_at": now()} if url else None,
        "usage": {"token_usage": raw.get("token_usage"), "runtime": raw.get("runtime")},
        "truncated": truncated,
        "result_id": result_id,
        "error": None,
    }

Next, the discovery tool. It uses the map agent, which collects URLs from a starting page. The limit field caps how many URLs come back.

python

def discover_pages(args: DiscoverInput) -> dict:
    body = {
        "url": str(args.start_url),
        "agent": "map",
        "maxDepth": 2,
        "maxPages": 3,
        "limit": args.max_urls,
        "maxRetries": 2,
    }
    if args.include_patterns:
        body["includePatterns"] = args.include_patterns
 
    raw = call_mrscraper(body)
    urls = (raw.get("data") or {}).get("urls", [])
    return envelope({"urls": urls, "count": len(urls)}, url=str(args.start_url), raw=raw)

Then the extraction tool. It turns the agent's field list into a prompt with a JSON schema, calls MrScraper and stores the full response.

python

import json
 
 
def extract_page(args: ExtractInput) -> dict:
    schema = {
        "type": "object",
        "properties": {
            f.name: {"type": [f.type, "null"], "description": f.description}
            for f in args.fields
        },
    }
    prompt = "Extract these fields from the page.\nJson Schema:\n" + json.dumps(schema)
 
    raw = call_mrscraper(
        {"url": str(args.url), "prompt": prompt, "agent": "general", "maxRetries": 2},
        html="false",
    )
 
    result_id = "r_" + uuid.uuid4().hex[:6]
    RESULT_STORE[result_id] = {"url": str(args.url), "retrieved_at": now(), "raw": raw}
 
    data = raw.get("data") or {}
    missing = [f.name for f in args.fields if data.get(f.name) is None]
    return envelope(
        {"fields": data, "missing_fields": missing},
        url=str(args.url),
        raw=raw,
        result_id=result_id,
    )

The missing_fields list matters. It tells the agent which fields the page did not provide, so it does not have to guess.

Finally, the stored-result tool. It returns the stored output in chunks.

python

CHUNK_CHARS = 4000
 
 
def get_stored_result(args: StoredResultInput) -> dict:
    stored = RESULT_STORE.get(args.result_id)
    if stored is None:
        raise ToolError("not_found", f"No stored result with id {args.result_id}.", retryable=False)
 
    raw = stored["raw"]
    text = raw.get("markdown") or raw.get("extractions") or json.dumps(raw.get("data") or {})
    chunk = text[args.offset: args.offset + CHUNK_CHARS]
    more = args.offset + CHUNK_CHARS < len(text)
 
    data = {"text": chunk, "next_offset": args.offset + CHUNK_CHARS if more else None}
    result = envelope(data, url=stored["url"], truncated=more, result_id=args.result_id)
    result["source"]["retrieved_at"] = stored["retrieved_at"]
    return result

Hand the result to your application

The agent works with short envelopes. Your application keeps the full records. When the agent finishes, your code can read RESULT_STORE (or your database) to save the extracted fields with their source URL and retrieval time.

How do you expose errors and limits?

A tool that fails quietly is worse than one that fails loudly. If a call returns empty data with ok: true, the agent may report "no information found" when the real problem was a timeout.

Return errors the agent can act on

Give every error three parts:

A code your app can count and test
A message written for the model, in plain words

A retryable flag, so the agent knows whether trying again makes sense

Code	When it happens	Retryable
`invalid_input`	The arguments failed validation	No, fix the arguments
`unauthorized`	MrScraper returned 401	No, your app must fix the token
`upstream_error`	MrScraper returned 500 or another server error	Yes
`timeout`	The request took too long	Yes
`not_found`	The `result_id` does not exist	No
`budget_exceeded`	Your app's limit for this session is used up	No

A 401 means your token is missing, invalid or expired. That is never something the agent can fix, so the message should tell it to stop. Our guide to 401 Unauthorized vs 403 Forbidden explains the difference between the two.

Wrap every tool the same way

python

from pydantic import ValidationError
 
INPUT_MODELS = {
    "discover_pages": (DiscoverInput, discover_pages),
    "extract_page": (ExtractInput, extract_page),
    "get_stored_result": (StoredResultInput, get_stored_result),
}
 
 
class ToolError(Exception):
    def __init__(self, code: str, message: str, retryable: bool):
        super().__init__(message)
        self.code, self.message, self.retryable = code, message, retryable
 
 
def failure(code: str, message: str, retryable: bool) -> dict:
    return {
        "ok": False, "data": None, "source": None, "usage": None,
        "truncated": False, "result_id": None,
        "error": {"code": code, "message": message, "retryable": retryable},
    }
 
 
def run_tool(name: str, arguments: dict, budget: dict) -> dict:
    if name not in INPUT_MODELS:
        return failure("invalid_input", f"Unknown tool {name}.", False)
    if budget["calls_left"] <= 0:
        return failure("budget_exceeded", "No tool calls left. Answer with what you have.", False)
 
    model, fn = INPUT_MODELS[name]
    try:
        args = model(**arguments)
    except ValidationError as e:
        return failure("invalid_input", f"Fix these arguments: {e.errors()[0]['msg']}", False)
 
    budget["calls_left"] -= 1
    try:
        return fn(args)
    except ToolError as e:
        return failure(e.code, e.message, e.retryable)
    except requests.Timeout:
        return failure("timeout", "The page took too long. You may retry once.", True)
    except requests.HTTPError as e:
        if e.response is not None and e.response.status_code == 401:
            return failure("unauthorized", "The data service rejected the credentials. Stop and report this.", False)
        return failure("upstream_error", "The data service had an error. You may retry once.", True)

Show limits before the agent hits them

Limits work best when the agent can see them. You can add the remaining budget to each result, like "calls_left": 4. The agent can then plan, instead of running out halfway through.

Two MrScraper request fields help on the API side:

maxRetries caps how many times MrScraper retries a request.
tokenCap caps the tokens a single scrape and its retries can use. Your app still owns the session limits, like total calls or total tokens per user. MrScraper reports usage per request, and your code adds it up.

How do you test tool selection?

Good descriptions only help if the model actually uses them. Test it.

Build a small test set

Write prompts where you know which tool should be called first. Include prompts that should not call any tool.

python

SELECTION_CASES = [
    {"prompt": "Which pages on https://docs.example.com cover webhooks?", "expected": "discover_pages"},
    {"prompt": "Get the plan names and monthly prices from https://example.com/pricing", "expected": "extract_page"},
    {"prompt": "Result r_7f3c9a was cut off. Show me the rest of the page text.", "expected": "get_stored_result"},
    {"prompt": "What does the word 'schema' mean?", "expected": None},
]

Score the first tool call

Run each prompt several times, since model output can vary. first_tool_call is your own model call. It sends the prompt and tool definitions, then returns the name of the first tool the model chose, or None.

python

from collections import Counter
 
 
def test_selection(first_tool_call, tools, runs=5):
    mistakes = Counter()
    correct = total = 0
    for case in SELECTION_CASES:
        for _ in range(runs):
            chosen = first_tool_call(case["prompt"], tools)
            total += 1
            if chosen == case["expected"]:
                correct += 1
            else:
                mistakes[(case["expected"], chosen)] += 1
    print(f"Correct: {correct}/{total}")
    for (expected, chosen), count in mistakes.most_common():
        print(f"  expected {expected}, got {chosen}: {count}")

Fix the description, then test again

When you see a pattern, change the tool description and rerun the test. For example, if the model calls extract_page on a homepage to "find" a pricing page, add a clearer "Do not use it to find pages" line.

Also test the error paths. Send a bad URL, a missing result_id and an empty budget. Check that each one returns the right error code and that the agent responds sensibly.

What about the MrScraper MCP server?

You do not always need to write your own tools. The MrScraper MCP server gives MCP clients three tools: scraperAPI for fetching page content, webSearch for search results and status for account usage.

Custom tools make sense when you want your own schema, result store or limits. The same design rules apply either way. Clear inputs, compact results and honest errors help the agent no matter where the tool comes from.

FAQs

How should tool inputs be defined?

Give each tool one job and a verb-based name. Describe when to use it and when not to. Use a typed schema with required fields, enums and limits. Keep timeouts, retries and credentials in your code, not in the model's inputs. Then check every call in code before it runs.

How do you test tool selection?

Write prompts where you know the right first tool, including prompts that need no tool. Run each one several times with your tool definitions and score how often the model picks correctly. Change the descriptions when you see a pattern of mistakes, then test again.

Which parts does MrScraper provide, and which parts must my application implement?

MrScraper is a developer-focused web data API. It discovers URLs from a starting page, fetches page content, returns Markdown or HTML and extracts structured JSON from a prompt and schema. Each response reports token usage and runtime. MrScraper also offers an MCP server with ready-made tools.

Your application owns the tool definitions, input checks, the result envelope, the result store, error mapping, session budgets and tool selection tests. Your team also decides which sources it has permission to collect from.

Next steps

Pick one tool your agent uses today. Rewrite its description with a clear "use this when" and "do not use this when." Add the error envelope. Then run a small selection test before and after.

To build the tools in this guide, see the MrScraper Web Scraper API and the API documentation. For a broader look at MCP, read Web Scraping MCP Server: Giving AI Agents Direct Access to Live Web Data.

Design Reliable Web Data Tools for AI Agents

How should tool inputs be defined?

Follow five rules

Write the tool definitions

Check inputs in code too

What belongs in structured tool results?

Use one result shape for every tool

Keep large content out of the result

Build the tools

Hand the result to your application

How do you expose errors and limits?

Return errors the agent can act on

Wrap every tool the same way

Show limits before the agent hits them

How do you test tool selection?

Build a small test set

Score the first tool call

Fix the description, then test again

What about the MrScraper MCP server?

FAQs

How should tool inputs be defined?

How do you test tool selection?

Which parts does MrScraper provide, and which parts must my application implement?

Next steps

Summarize this post

Take a Taste of Easy Scraping!

Find more insights here

Build an Agent Harness for Repeatable Web Research

Authenticate Web Data Tools for AI Agents Without Exposing Secrets

Build a Context Layer for an AI Agent Using Web Data