Context Engineering for Agents That Use Live Web Data

An agent that reads the web can give worse answers when you give it more data.

A full web page is full of menus, footers, cookie notices, and sections that have nothing to do with the question. The model has to sort through all of it. It may pick the wrong passage, mix up two plans, or lose track of where a fact came from.

Context engineering is the work of deciding what goes into the model's context window, in what shape, and what stays out. This guide shows how to do that for live web data, with a before and after example you can copy.

A before and after example

Here is the task:

Does Vendor X include single sign-on (SSO) on its Team plan?

Before: the whole page

The simple approach fetches the pricing page and pastes all of it into the prompt. The context now holds:

• The site menu and footer links
• Every plan, not just the Team plan
• A long FAQ about billing
• No record of when the page was collected
• No way to point to the exact sentence that answers the question The model might still answer correctly. But you cannot check the answer quickly, and you pay for every extra token.

After: a source packet

The engineered approach gives the model a small source packet instead. It holds only what the task needs, plus proof.

{
  "source_id": "S1",
  "url": "https://vendor.example.com/pricing",
  "retrieved_at": "2026-10-14T08:30:00+00:00",
  "fields": {
    "plan_name": "Team",
    "sso_included": false
  },
  "evidence": [
    {
      "heading": "Team",
      "text": "Includes shared workspaces, admin roles and audit logs. SSO is available on the Enterprise plan."
    }
  ]
}

Now the model sees the answer, the sentence that supports it, the page it came from and when it was collected. It can cite [S1]. You can check it in seconds.

What belongs in the context window?

A good rule: every item in the context should either help answer the current task or prove the answer.

Put these in

• The task and its rules. What the agent must do and how it should cite sources.
• Source packets. For each source: the URL, the retrieval time, the extracted fields and a few short evidence passages.
• Recent tool results, trimmed. Keep the parts the next step needs.

Leave these out

• Navigation menus, footers and cookie notices
• Raw HTML
• Sections that do not relate to the task
• Duplicate copies of the same page
• Old versions of a page, unless the task is to compare versions

How do you select source passages?

You can build source packets in four steps. MrScraper handles the first step. Your application handles the rest.

Step 1: Get clean page text

Use the MrScraper Scrape Markdown endpoint to turn a page into Markdown. Markdown keeps the headings, which you will use in the next step. For pages that load content in the browser, set browserRendering to true.

import os
import requests
 
MRSCRAPER_URL = "https://api.mrscraper.com/"
TOKEN = os.environ["MRSCRAPER_API_TOKEN"]
 
 
def fetch_markdown(url: str, render: bool = False) -> str:
    params = {"token": TOKEN, "markdown": "true"}
    if render:
        params["browserRendering"] = "true"
 
    response = requests.post(
        MRSCRAPER_URL,
        params=params,
        headers={"x-api-token": TOKEN, "Content-Type": "application/json"},
        json={"url": url, "maxRetries": 2},
        timeout=330,
    )
    response.raise_for_status()
    return response.json().get("markdown", "")

Step 2: Split the page by headings

Headings mark where one topic ends and the next begins. Splitting on them keeps each passage on a single topic.

import re
 
 
def split_by_headings(markdown: str) -> list[dict]:
    passages, heading, lines = [], "Top of page", []
    for line in markdown.splitlines():
        if re.match(r"^#{1,4} ", line):
            if lines:
                passages.append({"heading": heading, "text": "\n".join(lines).strip()})
            heading, lines = line.lstrip("#").strip(), []
        else:
            lines.append(line)
    if lines:
        passages.append({"heading": heading, "text": "\n".join(lines).strip()})
    return [p for p in passages if p["text"]]

Step 3: Score passages for the task

This example uses simple keyword counts so it is easy to follow. In production, you can swap in embeddings or a reranking model. The rest of the code stays the same.

def score(passage: dict, terms: list[str]) -> int:
    text = (passage["heading"] + " " + passage["text"]).lower()
    return sum(text.count(term.lower()) for term in terms)

Step 4: Build the packet

Keep only the top few passages and cap their length. Attach the URL and retrieval time.

from datetime import datetime, timezone
 
 
def build_packet(source_id, url, markdown, terms, fields,
                 max_passages=3, max_chars=800):
    ranked = sorted(split_by_headings(markdown),
                    key=lambda p: score(p, terms), reverse=True)
    evidence = [
        {"heading": p["heading"], "text": p["text"][:max_chars]}
        for p in ranked[:max_passages]
        if score(p, terms) > 0
    ]
    return {
        "source_id": source_id,
        "url": url,
        "retrieved_at": datetime.now(timezone.utc).isoformat(),
        "fields": fields,
        "evidence": evidence,
    }

Where do the fields come from? You have two options. You can ask your own model to read the evidence and fill them in. Or you can call the MrScraper Scrape JSON endpoint with a prompt and schema, and place its data in the packet.

How do you retain provenance through compaction?

Long agent sessions fill the context window. Many agents handle this by compacting: they replace older turns with a summary.

Summaries are where sources get lost. "Vendor X does not include SSO on Team" is easy to keep. "We learned this from the pricing page on October 14" often gets dropped.

Four rules that keep sources attached

1. Give every source an ID. Use short IDs like S1 and S2.
2. Store packets outside the context window. Keep a registry in your app that maps each ID to its full packet. The summary only needs the ID.
3. Require citations in every summary. Tell the model that each claim in a summary must end with a source ID.
4. Never paraphrase quotes. If the summary quotes a source, the quote must match the stored text word for word. A compacted summary then looks like this:

Vendor X's Team plan does not include SSO [S1].
SSO is listed under the Enterprise plan [S1].

Check summaries in code

Do not trust the model to follow the rules every time. Check each summary before it replaces the original turns.

def check_summary(summary: str, registry: dict) -> list[str]:
    problems = []
    for source_id in set(re.findall(r"\[(S\d+)\]", summary)):
        if source_id not in registry:
            problems.append(f"unknown source {source_id}")
    for sentence in re.split(r"(?<=[.!?])\s+", summary.strip()):
        if sentence and not re.search(r"\[S\d+\]", sentence):
            problems.append(f"no source: {sentence}")
    return problems

If the check finds problems, ask the model to try again or keep the original turns.

How do you test context quality?

Good context should lead to better answers that are easier to check. Test both.

Build a small test set

1. Write 20 to 50 questions your agent should answer.
2. For each question, record the correct answer and the page that proves it.
3. Store a copy of each page's Markdown so you can rerun tests on the same text.

Run the same questions two ways

Run the agent once with full pages and once with source packets. Then measure:

A quote check is quick to write:

def quote_matches(quote: str, page_markdown: str) -> bool:
    normalize = lambda s: " ".join(s.split())
    return normalize(quote) in normalize(page_markdown)

Run these tests whenever you change how you split, score or trim passages. That way you know whether each change helps.

FAQs

What belongs in the context window?

Only what helps the agent answer the current task or prove its answer. For web data, that usually means the task rules plus a source packet for each page: the URL, the retrieval time, the extracted fields and a few short evidence passages.

How do you test context quality?

Build a set of questions with known answers and known sources. Run your agent with different context setups. Measure correct answers, correct citations, exact quotes, token use and freshness.

Which parts does MrScraper provide, and which parts must my application implement?

MrScraper is a developer-focused web data API. It fetches pages, can capture content that loads in the browser, returns clean Markdown or HTML, and can extract structured JSON from a prompt and schema.

Context assembly is your application's job. That includes splitting and scoring passages, building source packets, the source registry, compaction rules and testing. Your team also decides which sources it has permission to use.