Evaluate AI Web Extraction With Field-Level Accuracy Tests

AI extraction turns a web page into a clean record: a title, a price, a list of specs. When it works, it saves hours of selector writing. But "it looks right" is not a test. You need to know how often each field is right, how often it is missing and how often the model fills in a value that is not on the page at all.

This guide builds a small, repeatable AI web extraction evaluation. You will create ground-truth records, score every field, handle nested records and report errors and cost in a way others can check.

We also ran the harness on a real sample, and the results are near the end. They are small and easy on purpose, so you can see how the method works before you point it at harder pages.

Why field-level scoring?

A single "accuracy" number hides the problems that matter. These four outcomes need different fixes:

Invented fields are the most dangerous. A missing price is easy to notice. A made-up price that looks real can flow into reports for weeks.

How do you create ground-truth records?

Ground truth is the correct record for each page, written down before you look at the extraction results. Your scores are only as good as these records.

Step 1: Pick and describe the sample

Choose 20 to 50 pages that look like the pages you will extract in production. Then write down exactly how you chose them, so others can repeat your test:

• The site or sites, and why you have permission to use them
• How you picked the pages, such as "every fifth product from the catalog listing"
• The date you fetched them
• What makes them easy or hard, such as layout differences or missing fields

Step 2: Freeze each page

Pages change. Save a copy of each page when you fetch it, and build ground truth from that copy. MrScraper's web fetching endpoint returns a page's raw HTML.

import hashlib
import json
import os
import requests
from datetime import datetime, timezone
 
MRSCRAPER_URL = "https://api.mrscraper.com/"
TOKEN = os.environ["MRSCRAPER_API_TOKEN"]
 
 
def freeze_page(url: str) -> dict:
    response = requests.get(
        MRSCRAPER_URL,
        params={"token": TOKEN, "url": url, "html": "true"},
        headers={"x-api-token": TOKEN},
        timeout=330,
    )
    response.raise_for_status()
    html = response.text
    return {
        "url": url,
        "retrieved_at": datetime.now(timezone.utc).isoformat(),
        "sha256": hashlib.sha256(html.encode()).hexdigest(),
        "html": html,
    }

Step 3: Write the correct record for each page

There are two good ways to do this:

• Read it from the page source. If a field always sits in the same tag, a small parser can read it. This is fast and objective, but it only works on sites with a stable layout.
• Label by hand. Have two people fill in each record on their own, then compare. Where they disagree, a third person decides. This works on any site. Either way, write null when the page truly has no value for a field. Without explicit nulls, you cannot score invented fields.

{
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "page_sha256": "…",
  "truth": {"title": "A Light in the Attic", "price": "£51.77", "availability": "In stock (22 available)"}
}

Step 4: Run the extraction you want to test

Here, that is the MrScraper Scrape JSON endpoint with a prompt and a JSON schema. Save every response, including the usage numbers you will need for the cost report.

SCHEMA = {
    "type": "object",
    "properties": {
        "title": {"type": ["string", "null"]},
        "price": {"type": ["string", "null"]},
        "availability": {"type": ["string", "null"]},
    },
}
 
 
def extract(url: str) -> dict:
    response = requests.post(
        MRSCRAPER_URL,
        params={"token": TOKEN, "html": "false"},
        headers={"x-api-token": TOKEN, "Content-Type": "application/json"},
        json={
            "url": url,
            "agent": "general",
            "prompt": "Extract the book title, price and availability text. "
                      "Use null for any field the page does not show.\n"
                      "Json Schema:\n" + json.dumps(SCHEMA),
            "maxRetries": 2,
        },
        timeout=330,
    )
    response.raise_for_status()
    result = response.json()
    return {"url": url, "data": result.get("data") or {},
            "token_usage": result.get("token_usage"), "runtime": result.get("runtime")}

How do you score missing and invented fields?

Compare each predicted field with the truth, after putting both into the same standard form. Then give each field one of five outcomes.

Normalize before you compare

"£51.77" and "51.77 GBP" may mean the same thing. Curly and straight apostrophes look the same to a reader but not to code. Write one small normalizer per field type, and keep it simple enough that anyone can read it.

import re
import unicodedata
 
 
def norm_text(v) -> str | None:
    if v is None:
        return None
    s = unicodedata.normalize("NFKC", str(v)).replace("’", "'").replace("‘", "'")
    return " ".join(s.split()).casefold() or None
 
 
def norm_money(v) -> str | None:
    if v is None:
        return None
    m = re.search(r"\d+(?:[.,]\d+)?", str(v))
    return m.group(0).replace(",", ".") if m else None
 
 
NORMALIZERS = {"title": norm_text, "price": norm_money, "availability": norm_text}

Be careful not to normalize away real errors. If your normalizer turns "In stock (2 available)" and "In stock (22 available)" into the same thing, it is too loose.

Give every field an outcome

def outcome(truth, predicted, normalize) -> str:
    t, p = normalize(truth), normalize(predicted)
    if t is None and p is None:
        return "correct_null"
    if t is None:
        return "invented"
    if p is None:
        return "missing"
    return "correct" if t == p else "wrong"
 
 
def score_record(truth: dict, predicted: dict) -> dict:
    return {f: outcome(truth.get(f), predicted.get(f), NORMALIZERS[f]) for f in NORMALIZERS}

Add up the results

Report four numbers for each field, plus one for whole records:

• Field accuracy: correct or correct-null, divided by all pages
• Missing rate: missing, divided by pages where the truth has a value
• Invented rate: invented, divided by pages where the truth is null
• Wrong rate: wrong, divided by pages where both have a value
• Record exact match: pages where every field is correct

from collections import Counter
 
 
def summarize(scored: list[dict]) -> dict:
    fields = list(NORMALIZERS)
    report = {}
    for f in fields:
        c = Counter(s[f] for s in scored)
        has_truth = c["correct"] + c["wrong"] + c["missing"]
        truth_null = c["correct_null"] + c["invented"]
        report[f] = {
            "accuracy": (c["correct"] + c["correct_null"]) / len(scored),
            "missing_rate": c["missing"] / has_truth if has_truth else None,
            "invented_rate": c["invented"] / truth_null if truth_null else None,
            "wrong": c["wrong"],
        }
    exact = sum(all(s[f] in ("correct", "correct_null") for f in fields) for s in scored)
    report["record_exact_match"] = exact / len(scored)
    return report

If your sample has no pages where a field is truly null, the invented rate cannot be measured. Report it as "not tested," not as zero. Then add pages that are missing that field.

How do you evaluate nested records?

Many real extractions return lists of objects: product variants, spec rows, authors, menu items. You cannot compare those field by field until you know which predicted item matches which true item.

Step 1: Match items by a key

Pick a field that identifies each item, like a SKU or a spec name. Match predicted items to true items on that key. Unmatched true items are missing items. Unmatched predicted items are invented items.

Step 2: Score matched items field by field

Use the same outcome function on each sub-field of each matched pair.

def score_list(truth_items: list[dict], predicted_items: list[dict], key: str,
               sub_normalizers: dict) -> dict:
    t_by_key = {norm_text(i.get(key)): i for i in truth_items}
    p_by_key = {norm_text(i.get(key)): i for i in predicted_items}
    matched = set(t_by_key) & set(p_by_key)
 
    sub = Counter()
    for k in matched:
        for f, normalize in sub_normalizers.items():
            sub[outcome(t_by_key[k].get(f), p_by_key[k].get(f), normalize)] += 1
 
    return {
        "item_precision": len(matched) / len(p_by_key) if p_by_key else None,
        "item_recall": len(matched) / len(t_by_key) if t_by_key else None,
        "missing_items": sorted(set(t_by_key) - matched),
        "invented_items": sorted(set(p_by_key) - matched),
        "sub_field_outcomes": dict(sub),
    }

For example, a spec table might be scored with key="name" and sub_normalizers={"value": norm_text}.

Decide whether order matters

For some lists, order is part of the truth, such as steps in a recipe. For others, it is not, such as a set of tags. Write down which rule you use for each list field. If order matters, add a check that compares the order of the matched keys.

How do you report errors and cost?

Report every error with enough detail to fix it, and report what the test cost to run.

Keep an error log

Save one row for every field that was not correct. The rows below are examples of the format:

Then read through it. Errors often cluster. If every wrong price comes from pages with a sale price, the fix is a clearer field description, not a new tool.

Report cost and time

Each Scrape JSON response includes token_usage and runtime. Report the average per page, and convert tokens to money with your MrScraper plan's rate. If you run a second method for comparison, report its cost the same way, including any retries.

Publish the full setup

Anyone reading your results should be able to repeat them. List the sample description, date, schema, prompt, normalizers, number of pages and any pages you dropped and why.

What we measured

We ran this harness on a small sample to show the method in action.

The setup

• Date: October 21, 2026
• Pages: 20 book product pages from books.toscrape.com, a practice site built for scraping, picked at even spacing across its catalog
• Fields: title, price and availability, all flat strings
• Extraction: MrScraper Scrape JSON with a three-field schema and the prompt "Extract the book title, price and availability text."
• Ground truth: read from each page's raw HTML with a small parser

The results

All 20 records were exact matches on every field. Each Scrape JSON call used 10 MrScraper tokens.

What this does and does not show

This was an easy test. The site has one clean layout, every field is plain visible text and every page has every field. A perfect score here shows that the harness works and that this kind of field is easy to extract. It does not tell you how extraction will do on your pages.

It also leaves two things untested. With no null fields in the truth, there was no chance to invent a value, so the invented rate is not measured. And with only flat fields, nested scoring was not used. A stronger next test would add fields that some pages lack, a nested list such as the product information table and pages from more than one site.

Hand off to your application

Keep the frozen pages, truth records, predictions and error log together, with the date. When you change your prompt, schema or extraction settings, rerun the same test on the same frozen pages. Then you can see whether the change helped, instead of guessing.

FAQs

How do you create ground-truth records?

Pick a sample of pages, describe how you chose them and save a copy of each page. Then write the correct record for each one, either with a parser on the saved source or by having two people label each page. Use explicit null for fields the page does not have.

How do you report errors and cost?

Keep a log row for every field that was not correct, with the URL, truth, prediction and outcome. Report missing, invented and wrong rates per field, plus record exact match. Add the average tokens and runtime per page from each extraction response, and list the full setup so others can repeat it.

Which parts does MrScraper provide, and which parts must my application implement?

MrScraper is a developer-focused web data API. In this guide, it fetches raw page HTML for ground truth and extracts structured records with Scrape JSON. Each Scrape JSON response reports its token usage and runtime.

Your application builds the evaluation: the sample, frozen pages, truth records, normalizers, scoring, nested matching and the error log. Your team also decides which pages it has permission to use.