Create a Source-Verified Question-and-Answer Dataset From Web Pages

A question-and-answer dataset is useful for many things: testing a support bot, checking a search system or measuring whether a model really understands your product. It is only useful if the answers are right.

The fastest way to build one is to have a model read your pages and write questions. The risk is that the model also writes answers the page never states. Those answers look just as convincing as the real ones.

This guide builds a Q&A dataset from web pages where every answer is an exact span of text from its source page, with the URL attached. Questions whose answers cannot be found word for word are rejected before a person ever reviews them. The example uses product documentation, which is non-sensitive and owned by the team that publishes it.

What you will build

Each item in the finished dataset looks like this:

{
  "id": "q_0142",
  "question": "How many seconds does the Scrape JSON endpoint wait for a page to load by default?",
  "answer": "300",
  "passage_id": "scrape-json#query-parameters",
  "answer_start": 412,
  "url": "https://docs.mrscraper.com/docs/api/scraper-api/scrape-json",
  "retrieved_at": "2026-10-22T04:10:00+00:00",
  "content_sha256": "…",
  "review": "approved"
}

The question here is based on a real fact from MrScraper's own API docs: the timeout parameter defaults to 300 seconds. The offset and hash are shown as examples of the format.

How do you choose a non-sensitive source corpus?

Start with pages you own, that contain no personal data and that state facts clearly. Product documentation is a near-perfect fit.

Good and poor sources

Facts that change, like prices and limits, are fine as long as you record the retrieval date and plan to refresh the dataset.

Collect the pages

Fetch each page with the MrScraper Scrape Markdown endpoint. Markdown keeps headings, which you will use to split pages into passages.

import hashlib
import os
import re
import requests
from datetime import datetime, timezone
 
MRSCRAPER_URL = "https://api.mrscraper.com/"
TOKEN = os.environ["MRSCRAPER_API_TOKEN"]
 
 
def fetch_page(url: str) -> dict:
    response = requests.post(
        MRSCRAPER_URL,
        params={"token": TOKEN, "markdown": "true"},
        headers={"x-api-token": TOKEN, "Content-Type": "application/json"},
        json={"url": url, "maxRetries": 2},
        timeout=330,
    )
    response.raise_for_status()
    md = response.json().get("markdown", "")
    return {
        "url": url,
        "retrieved_at": datetime.now(timezone.utc).isoformat(),
        "sha256": hashlib.sha256(md.encode()).hexdigest(),
        "markdown": md,
    }

Before you use a page, check that it is the page you asked for. A fetch can return an error page without raising an error, and an error page makes a poor source of facts.

Split pages into passages

Each question should be answerable from one short passage. Split on headings, and give each passage a stable ID made from the page and heading.

def passages(page: dict, max_chars: int = 2500) -> list[dict]:
    out, heading, lines = [], "intro", []
    slug = page["url"].rstrip("/").rsplit("/", 1)[-1]
 
    def flush():
        text = "\n".join(lines).strip()
        if len(text) >= 200:
            pid = f"{slug}#{re.sub(r'[^a-z0-9]+', '-', heading.lower()).strip('-')}"
            out.append({"passage_id": pid, "heading": heading, "text": text[:max_chars],
                        "url": page["url"], "retrieved_at": page["retrieved_at"],
                        "content_sha256": page["sha256"]})
 
    for line in page["markdown"].splitlines():
        m = re.match(r"^#{1,3} (.+)", line)
        if m:
            flush()
            heading, lines = m.group(1).strip(), []
        else:
            lines.append(line)
    flush()
    return out

Very short passages are skipped. They rarely hold enough for a good question.

How do you generate questions supported by the text?

Ask a model to write questions about one passage at a time. Ask it to copy each answer word for word from the passage. Your code will check that it did.

The prompt

generate is your own model call. It sends this prompt and returns the model's JSON reply.

Read the passage below and write up to 3 questions a user of this product might ask.
 
Rules:
- Each answer must be copied exactly from the passage. Do not reword it.
- Keep answers short: a value, a name or one phrase.
- Each question must make sense on its own. Do not write "this page" or "the passage".
- Do not include the answer in the question.
- Skip the passage if it has no clear facts.
 
Return JSON: {"items": [{"question": "...", "answer": "..."}]}
 
Passage (from {url}, section "{heading}"):
{text}

Why ask for copied answers

A copied answer can be checked with code. A rewritten answer can only be checked by a person or another model. Asking for exact copies turns most of the review work into a simple string search.

How do you reject unsupported answers?

Run every generated item through a set of checks. Each check has a clear reason, and every rejection is logged.

Check 1: The answer must appear in the passage

Find the answer in the passage text. Allow for differences in spacing only. If it is not there, reject it.

def find_span(answer: str, text: str) -> int | None:
    pattern = r"\s+".join(re.escape(part) for part in answer.split())
    m = re.search(pattern, text) if pattern else None
    return m.start() if m else None

Check 2: The question must be fair

Reject questions that give away their answer, depend on missing context or are too long to be useful. A one-word answer like "screenshot" can fairly appear in a question like "Which field returns a screenshot?", so the give-away check only applies to numbers and multi-word answers. Check 3 catches the rest.

VAGUE = re.compile(r"\b(this|the) (page|passage|section|document|article)\b", re.I)
 
 
def gives_away(q: str, answer: str) -> bool:
    # A number or a multi-word answer inside the question gives it away.
    # A single plain word often repeats a word from the question, which is fine.
    if not (re.search(r"\d", answer) or len(answer.split()) > 1):
        return False
    return re.search(rf"(?<!\w){re.escape(answer.lower())}(?!\w)", q.lower()) is not None
 
 
def question_problems(q: str, answer: str) -> list[str]:
    problems = []
    if gives_away(q, answer):
        problems.append("question contains the answer")
    if VAGUE.search(q):
        problems.append("question depends on missing context")
    if len(answer.split()) > 15:
        problems.append("answer is too long")
    if not q.strip().endswith("?"):
        problems.append("not phrased as a question")
    return problems

Check 3: A second, blind answer must agree

An answer can be a real span and still be the wrong answer to the question. To catch this, ask a model the question with the passage, but without showing it the expected answer. If its answer does not contain the expected span, or the expected span does not contain its answer, send the item to review.

answer_blind is your own model call. Use a separate request, so the second answer is not influenced by the first.

def norm(s: str) -> str:
    return " ".join(s.lower().split()).strip(" .")
 
 
def blind_agrees(question: str, passage: str, expected: str, answer_blind) -> bool:
    got = norm(answer_blind(question, passage))
    exp = norm(expected)
    return bool(got) and (exp in got or got in exp)

Check 4: Remove duplicate questions

Different passages often lead to the same question. Keep the first one and drop near-copies.

def words(s: str) -> set:
    return set(re.findall(r"\w+", s.lower()))
 
 
def is_duplicate(q: str, kept: list[str], threshold: float = 0.8) -> bool:
    w = words(q)
    return any(len(w & words(k)) / max(1, len(w | words(k))) >= threshold for k in kept)

Put the checks together

def build_items(passage_list: list[dict], generate, answer_blind) -> tuple[list, list]:
    kept, rejected, kept_questions = [], [], []
    for p in passage_list:
        for item in generate(p).get("items", []):
            q, a = item.get("question", "").strip(), item.get("answer", "").strip()
            reasons = question_problems(q, a)
            start = find_span(a, p["text"]) if a else None
            if start is None:
                reasons.append("answer not found in passage")
            if is_duplicate(q, kept_questions):
                reasons.append("duplicate question")
            record = {"question": q, "answer": a, "passage_id": p["passage_id"],
                      "answer_start": start, "url": p["url"], "retrieved_at": p["retrieved_at"],
                      "content_sha256": p["content_sha256"]}
            if reasons:
                rejected.append({**record, "reasons": reasons})
                continue
            record["review"] = "pending" if blind_agrees(q, p["text"], a, answer_blind) else "needs_check"
            kept.append(record)
            kept_questions.append(q)
    return kept, rejected

Items that pass every check start as pending. Items where the blind answer disagreed start as needs_check, so reviewers see them first.

How do you export a reviewed dataset?

Have people review the items, record each decision and export only approved items. Split by page, so questions from the same page never land in both training and testing.

Review

Give reviewers a simple table: the question, the answer, the passage with the answer highlighted and the source URL. Ask for one decision per item:

• Approve: the question is clear and the answer is right
• Edit question: the answer is right, but the question needs rewording
• Reject: the question is unclear, trivial or misleading Reviewers may edit questions, but not answers. If an answer needs changing, reject the item. That keeps every approved answer an exact span from the source.

Review all needs_check items and a random sample of pending items. If the sample shows many problems, review everything.

Export

import json
 
 
def split_of(url: str) -> str:
    bucket = int(hashlib.sha256(url.encode()).hexdigest(), 16) % 10
    return "test" if bucket < 2 else "train"
 
 
def export(items: list[dict]) -> None:
    files = {s: open(f"qa_{s}.jsonl", "w", encoding="utf-8") for s in ("train", "test")}
    for n, item in enumerate(i for i in items if i["review"] == "approved"):
        row = {"id": f"q_{n:04d}", **item}
        files[split_of(item["url"])].write(json.dumps(row, ensure_ascii=False) + "\n")
    for f in files.values():
        f.close()

Ship a short dataset card with the files. Say which site and pages were used, the collection date, which model generated the questions, how many items were generated, rejected, reviewed and approved, and the main rejection reasons.

Keep it fresh

Docs change. Store the content_sha256 of each source page. When you refetch a page and its hash changes, mark its items as stale. Then rerun the span check against the new text. Items whose answers are no longer in the page go back to review.

FAQs

How do you choose a non-sensitive source corpus?

Use pages you own that contain no personal data and state facts clearly, such as your API docs, guides and help center. Avoid forums, pages behind a login and pages where a wrong answer could cause harm, such as medical or legal advice. Record the retrieval date so you can refresh changing facts.

How do you export a reviewed dataset?

Have reviewers approve, edit the question or reject each item, without changing answers. Export only approved items as JSON Lines, with the source URL, passage ID, answer position, retrieval date and content hash. Split by page so one page's questions never appear in both training and test files, and include a dataset card.

Which parts does MrScraper provide, and which parts must my application implement?

MrScraper is a developer-focused web data API. In this guide, it fetches each documentation page and returns it as Markdown, with the headings needed to split it into passages.

Question generation and validation are your own application logic. That includes the passages, prompts, model calls, span checks, blind answer checks, duplicate removal, review, export and freshness checks. Your team also decides which pages it has permission to use.