Prepare Web Data for Fine-Tuning: Quality, Provenance and Splits

Fine-tuning teaches a model from examples. If the examples are messy, duplicated or taken from pages you had no right to use, the model learns the mess and you inherit the risk.

Web pages can be a good source of training examples, but only after careful preparation. This guide walks through that work for a concrete task. You will choose records that fit the task, filter duplicates and weak examples, keep a record of where every example came from and who allowed its use and split the data so your test results are honest.

MrScraper collects the pages. It does not train models. Everything after collection in this guide is your own code, and the training itself happens with whichever model provider or framework you choose.

The example task

Train a small model to route customer questions to the right product area of your help center.

The training data comes from your own help center. Each article has a title, which often reads like a customer question, and a category, which is the product area. For example:

The title becomes the input, and the category becomes the label. Both are copied straight from the page, so no model writes or rewrites any training text.

Which web records fit your training task?

A record fits when it looks like what the model will see in real use, has a label you can trust and comes from a source you are allowed to use for training. If any of those three is missing, leave it out.

Check fit before you collect

The second row matters more than it looks. If customers type short questions, train on short, question-like text. A model trained only on long articles may do poorly on one-line questions, even if every article is perfect.

Collect the pages

Many sites publish a sitemap listing every page. MrScraper's web fetching endpoint can fetch it, and your code can pick out the article URLs. Then the Scrape Markdown endpoint returns each article as Markdown, which keeps the title and the breadcrumb that holds the category.

import hashlib
import os
import re
import requests
from datetime import datetime, timezone
 
MRSCRAPER_URL = "https://api.mrscraper.com/"
TOKEN = os.environ["MRSCRAPER_API_TOKEN"]
SITEMAP = "https://help.example.com/sitemap.xml"   # Your own help center
ARTICLE = re.compile(r"^https://help\.example\.com/articles/[^/]+/?$")
 
 
def fetch_raw(url: str) -> str:
    response = requests.get(MRSCRAPER_URL, params={"token": TOKEN, "url": url, "html": "true"},
                            headers={"x-api-token": TOKEN}, timeout=330)
    response.raise_for_status()
    return response.text
 
 
def article_urls() -> list[str]:
    locs = re.findall(r"<loc>\s*(.*?)\s*</loc>", fetch_raw(SITEMAP))
    return sorted({u for u in locs if ARTICLE.match(u)})
 
 
def fetch_article(url: str) -> dict:
    response = requests.post(
        MRSCRAPER_URL,
        params={"token": TOKEN, "markdown": "true"},
        headers={"x-api-token": TOKEN, "Content-Type": "application/json"},
        json={"url": url, "maxRetries": 2},
        timeout=330,
    )
    response.raise_for_status()
    md = response.json().get("markdown", "")
    title = re.search(r"^# (.+)$", md, re.M)
    crumbs = re.findall(r"^\s*\* \[([^\]]+)\]\(", md, re.M)
    return {
        "url": url,
        "retrieved_at": datetime.now(timezone.utc).isoformat(),
        "sha256": hashlib.sha256(md.encode()).hexdigest(),
        "title": title.group(1).strip() if title else None,
        "category": crumbs[-1] if crumbs else None,
        "body": md,
    }

The breadcrumb pattern depends on your site's layout. Check it on a few pages before you run it on all of them.

How do you filter duplicates and low-quality examples?

Run every record through a set of checks, and write down why each one was dropped. The drop log matters as much as the kept data, because it shows what your dataset leaves out.

Quality checks

EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
PHONE = re.compile(r"\+?\d[\d\s().-]{8,}\d")
OK_CONTACTS = {"support@example.com"}   # Your own public support addresses
 
 
def quality_problems(r: dict, label_counts: dict, min_per_label: int = 20) -> list[str]:
    problems = []
    if not r["title"] or not r["category"]:
        problems.append("missing title or category")
    if len(r["body"]) < 300:
        problems.append("page too short to be a real article")
    if "can't be reached" in r["body"] or "can’t be reached" in r["body"]:
        problems.append("fetch returned an error page")
    if r["title"] and r["title"].lower() not in r["body"].lower():
        problems.append("title not found in page text")
    emails = set(EMAIL.findall(r["body"])) - OK_CONTACTS
    if emails or PHONE.search(r["body"]):
        problems.append("possible personal contact details: send for review")
    if r["category"] and label_counts.get(r["category"], 0) < min_per_label:
        problems.append("category has too few examples")
    return problems

Each check catches a real problem:

• Missing fields usually mean the page is not an article, or the breadcrumb pattern did not match.
• Very short pages are often redirects, stubs or "page moved" notices.
• Error pages can come back from a fetch that technically succeeded. In one of our own test runs, a fetch returned a browser "This site can't be reached" page with no error raised. Without a check, it would have been training data.
• Title not in body suggests the fetch returned a different page than expected.
• Contact details can be a sign the page holds personal data. Your own public support address is fine, so list it as allowed. Send anything else for review before it goes into training, even from your own help center. The phone pattern is loose on purpose, so expect some false alarms, such as long version numbers.
• Rare categories cannot be learned from three examples. Merge them into a broader label or leave them out.

Remove duplicates

Help centers often repeat the same article under different URLs, or keep near-copies for different product versions. Duplicates make a label look more common than it is, and if one copy lands in training and another in testing, your test score is inflated.

Catch exact copies with a hash of the normalized text. Catch near-copies by comparing overlapping word groups, called shingles.

def shingles(text: str, n: int = 5) -> set:
    words = re.findall(r"\w+", text.lower())
    return {" ".join(words[i:i + n]) for i in range(max(1, len(words) - n + 1))}
 
 
def jaccard(a: set, b: set) -> float:
    return len(a & b) / len(a | b) if a | b else 0.0
 
 
def duplicate_groups(records: list[dict], threshold: float = 0.8) -> dict:
    """Give near-duplicate records the same group ID."""
    sigs = [shingles(r["body"]) for r in records]
    group = {}
    for i, r in enumerate(records):
        if r["url"] in group:
            continue
        group[r["url"]] = r["url"]
        for j in range(i + 1, len(records)):
            other = records[j]["url"]
            if other not in group and jaccard(sigs[i], sigs[j]) >= threshold:
                group[other] = r["url"]
    return group

This compares every pair, which is fine for a few thousand pages. For much larger sets, use a library that does approximate matching, such as MinHash.

Keep one record per group for training, and use the group ID again when you split the data.

How do you preserve source and usage permissions?

Keep a manifest: one row per record that says where it came from, what the page looked like and why you are allowed to use it. Store it next to the dataset and ship it with every version.

What goes in the manifest

{
  "record_id": "a3f9c2e1",
  "url": "https://help.example.com/articles/rotate-api-token",
  "retrieved_at": "2026-10-22T03:15:00+00:00",
  "content_sha256": "…",
  "source_owner": "Example Inc.",
  "permission_basis": "First-party content owned by Example Inc.",
  "allowed_uses": ["fine_tuning", "evaluation"],
  "reviewed_by": "content-legal@example.com",
  "reviewed_on": "2026-10-20",
  "dropped": false,
  "drop_reasons": []
}

Three fields deserve care:

• permission_basis says why you can use the page. "We own it" is the simplest answer. For anything else, such as partner content or licensed material, get a qualified review and record the outcome. Being able to reach a page does not mean you may train on it.
• allowed_uses limits where the record can go. A page cleared for evaluation might not be cleared for training.
• content_sha256 lets you prove exactly which version of a page you used, even after the page changes.

Make removals possible

If a page is later withdrawn or a permission changes, you need to find every record that came from it. The manifest makes that a lookup, not a search. Remove the records, rebuild the splits and note the change in your dataset's change log.

How do you prevent train/test leakage?

Leakage happens when the test set contains examples the model already saw in training, or near-copies of them. The test score then looks better than real-world performance will be. Prevent it by splitting groups, not rows.

Split by group, not by row

Put all members of a duplicate group, and all records from the same page, on the same side of the split. Then assign whole groups to train, validation or test with a stable hash, so the same group always lands in the same split.

def split_of(group_id: str, test: float = 0.15, val: float = 0.15) -> str:
    bucket = int(hashlib.sha256(group_id.encode()).hexdigest(), 16) % 1000 / 1000
    if bucket < test:
        return "test"
    if bucket < test + val:
        return "validation"
    return "train"

A stable hash means that adding new pages later does not move old ones between splits. That keeps your test set comparable over time.

Check the result

After splitting, run two checks:

1. Label balance. Every category should appear in every split. If a small category ends up only in training, merge it or rebalance.
2. Cross-split overlap. Compare each test record with the training records using the shingle check above. Any pair above your threshold is a leak. Move or drop it.

def leaks(train: list[dict], test: list[dict], threshold: float = 0.8) -> list[tuple]:
    train_sigs = [(r["url"], shingles(r["title"] + " " + r["body"])) for r in train]
    found = []
    for r in test:
        sig = shingles(r["title"] + " " + r["body"])
        for url, tsig in train_sigs:
            if jaccard(sig, tsig) >= threshold:
                found.append((r["url"], url))
    return found

Consider a time split too

If your help center changes over time, also hold out the newest articles as a second test set. It shows how the model handles topics it has never seen, which is closer to real use than a random split.

Export the dataset

Write each split to its own file. Keep the training examples simple, and keep the provenance in the manifest, linked by record_id.

import json
 
 
def export(records: list[dict], groups: dict) -> None:
    files = {s: open(f"{s}.jsonl", "w", encoding="utf-8") for s in ("train", "validation", "test")}
    for r in records:
        if groups[r["url"]] != r["url"]:
            continue   # Keep one record per duplicate group
        rid = hashlib.sha256(r["url"].encode()).hexdigest()[:8]
        example = {"record_id": rid, "input": r["title"], "label": r["category"]}
        files[split_of(groups[r["url"]])].write(json.dumps(example, ensure_ascii=False) + "\n")
    for f in files.values():
        f.close()

Each training provider expects its own file format, such as a specific chat message layout. Convert these neutral files to that format in a final step, following your provider's current fine-tuning docs.

Add a short dataset card with the files. It should say what the data is for, where it came from, when it was collected, what was filtered out and why, how it was split and what it should not be used for.

FAQs

Which web records fit your training task?

Records fit when their inputs look like what the model will see in real use, their labels can be trusted and you are allowed to use them for training. Your own help articles, docs and product pages are the safest start. Skip pages with personal data, index pages and anything you lack clear rights to.

How do you prevent train/test leakage?

Group near-duplicates and records from the same page, then assign whole groups to train, validation or test with a stable hash. After splitting, check that every label appears in each split and that no test record closely matches a training record. A separate test set of the newest pages adds a further check.

Which parts does MrScraper provide, and which parts must my application implement?

MrScraper is a developer-focused web data API. In this guide, it fetches your sitemap and returns each article as Markdown. It collects data. It does not train models.

Your application handles everything after collection: fit rules, quality checks, duplicate groups, the provenance manifest, splits, leakage checks and export. Your team, with qualified review where needed, decides which pages you have the right to use for training.