Benchmark Heading-Aware and Semantic Chunking for Web Documentation

How you split documents into chunks changes what your retrieval system can find. Split in the wrong place, and the answer to a question ends up cut in half across two chunks. Neither half matches the question well, so neither gets retrieved.

Two popular ways to split documentation pages are:

• Heading-aware chunking, which follows the page's own headings
• Semantic chunking, which uses embeddings to split where the topic seems to change Blog posts often say one is better. The honest answer is that it depends on your pages and your questions. This guide shows you how to test both on your own documentation, with metrics you can trust and an error review that explains the numbers.

Our RAG pipeline guide covers recursive character splitting. This article is about comparing two other strategies side by side.

Set up a fair experiment

A chunking test is only fair if everything except the chunker stays the same. Fix these before you start:

Freeze the page text first

Pages change. If you fetch them again halfway through the test, your two strategies may see different text. So fetch each page once, save it with its URL and retrieval time, and run both chunkers on that saved copy.

The MrScraper Scrape Sitemap endpoint collects URLs from a starting page. The Scrape Markdown endpoint returns each page as Markdown, which keeps the headings both chunkers need.

Use documentation your team owns or has permission to use.

import json
import os
import requests
from datetime import datetime, timezone
 
MRSCRAPER_URL = "https://api.mrscraper.com/"
TOKEN = os.environ["MRSCRAPER_API_TOKEN"]
 
 
def call_mrscraper(body: dict, **params) -> dict:
    response = requests.post(
        MRSCRAPER_URL,
        params={"token": TOKEN, **params},
        headers={"x-api-token": TOKEN, "Content-Type": "application/json"},
        json={**body, "maxRetries": 2},
        timeout=330,
    )
    response.raise_for_status()
    return response.json()
 
 
def build_corpus(start_url: str, pattern: str, max_pages: int = 30) -> list[dict]:
    found = call_mrscraper({
        "url": start_url, "agent": "map", "maxDepth": 2,
        "maxPages": 5, "limit": max_pages, "includePatterns": [pattern],
    })
    corpus = []
    for url in (found.get("data") or {}).get("urls", [])[:max_pages]:
        page = call_mrscraper({"url": url}, markdown="true")
        corpus.append({
            "url": url,
            "retrieved_at": datetime.now(timezone.utc).isoformat(),
            "markdown": page.get("markdown", ""),
        })
    with open("corpus.json", "w") as f:
        json.dump(corpus, f)
    return corpus

From here on, every step reads corpus.json. Nothing else calls the web.

How do headings define document sections?

Documentation writers already split their pages into sections. A heading like ## Rotate an API key tells you exactly what the text below it covers. Heading-aware chunking uses that structure.

The rules

1. Start a new section at each heading, up to level 3.
2. Keep track of the heading path, such as Authentication > API keys > Rotate a key.
3. Ignore # lines inside code blocks. They are comments, not headings.
4. If a section is too long, split it at paragraph breaks, and keep the heading path on each piece.

The code

import re
 
FENCE = "`" * 3   # A Markdown code fence
 
 
def heading_chunks(page: dict, max_chars: int = 1500) -> list[dict]:
    chunks, path, lines, in_code = [], [], [], False
 
    def flush():
        text = "\n".join(lines).strip()
        if not text:
            return
        pieces, current = [], ""
        for para in text.split("\n\n"):
            if current and len(current) + len(para) > max_chars:
                pieces.append(current)
                current = ""
            current = f"{current}\n\n{para}" if current else para
        pieces.append(current)
        for piece in pieces:
            chunks.append({"url": page["url"], "path": " > ".join(path), "text": piece})
 
    for line in page["markdown"].splitlines():
        if line.strip().startswith(FENCE):
            in_code = not in_code
        match = None if in_code else re.match(r"^(#{1,3}) (.+)", line)
        if match:
            flush()
            lines = []
            level = len(match.group(1))
            path = path[: level - 1] + [match.group(2).strip()]
        else:
            lines.append(line)
    flush()
    return chunks

When you embed a heading chunk, add the path to the front of the text, like "Authentication > API keys > Rotate a key\n\n...". That gives short sections the context they would otherwise lack. Treat this as a setting you test, not a given. You can run the heading strategy both with and without the path.

How does semantic splitting change those sections?

Semantic chunking ignores the page's headings. It looks at the meaning of each sentence instead.

The rules

1. Split the page into sentences. Keep each code block whole.
2. Embed every sentence.
3. Measure how similar each sentence is to the next one.
4. Start a new chunk where similarity drops sharply. A common rule is "the lowest 20% of drops."
5. Enforce a minimum and maximum size, so you do not get one-sentence chunks or huge ones.

The code

embed is your own function. It takes a list of strings and returns a list of vectors from your embedding model. Use the same model you will use for retrieval.

import math
 
 
def cosine(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    norm = math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b))
    return dot / norm if norm else 0.0
 
 
def split_units(markdown: str) -> list[str]:
    units = []
    code_block = f"({FENCE}.*?{FENCE})"
    for block in re.split(code_block, markdown, flags=re.S):
        if block.startswith(FENCE):
            units.append(block)
        else:
            text = re.sub(r"^#{1,6} ", "", block, flags=re.M)
            units += [s.strip() for s in re.split(r"(?<=[.!?])\s+|\n{2,}", text) if s.strip()]
    return units
 
 
def semantic_chunks(page: dict, embed, break_pct: float = 0.2,
                    min_chars: int = 200, max_chars: int = 1500) -> list[dict]:
    units = split_units(page["markdown"])
    if not units:
        return []
    vectors = embed(units)
    sims = [cosine(vectors[i], vectors[i + 1]) for i in range(len(units) - 1)]
    cutoff = sorted(sims)[int(len(sims) * break_pct)] if sims else 0.0
 
    chunks, current = [], units[0]
    for i, unit in enumerate(units[1:]):
        topic_break = sims[i] <= cutoff and len(current) >= min_chars
        too_long = len(current) + len(unit) > max_chars
        if topic_break or too_long:
            chunks.append(current)
            current = unit
        else:
            current = f"{current} {unit}"
    chunks.append(current)
    return [{"url": page["url"], "path": "", "text": c} for c in chunks]

Notice two costs that heading chunking does not have. Semantic chunking embeds every sentence once just to decide where to split, before you embed the chunks themselves. And its results depend on settings like break_pct, so you should test a few values.

How do you build a retrieval test set?

The test set matters more than the code. A weak set makes both strategies look the same.

Write questions with exact answer quotes

Each test item has a question, the page it is answered on and a short quote copied word for word from the saved Markdown of that page.

{
  "id": "t07",
  "question": "How long does an API token stay valid?",
  "url": "https://docs.example.com/auth/tokens",
  "answer_quote": "You choose an expiration date when you create the token.",
  "type": "single_fact"
}

The quote is what makes scoring objective. A chunk either contains it or it does not.

Mix question types

Different questions stress chunkers in different ways. Aim for 50 to 100 items across these types:

Draft with help, check by hand

A model can draft questions and quotes from your pages. A person should check every one. Then confirm each quote really appears on its page:

def normalize(text: str) -> str:
    return " ".join(text.split()).lower()
 
 
def check_test_set(items: list[dict], corpus: list[dict]) -> list[str]:
    pages = {p["url"]: normalize(p["markdown"]) for p in corpus}
    problems = []
    for item in items:
        if item["url"] not in pages:
            problems.append(f"{item['id']}: page not in corpus")
        elif normalize(item["answer_quote"]) not in pages[item["url"]]:
            problems.append(f"{item['id']}: quote not found on page")
    return problems

Keep answer quotes short, one or two sentences. Long quotes are more likely to cross a chunk boundary under any strategy, which hides the real differences.

Run the benchmark

Both strategies go through the same retrieval code and the same scoring.

Score each question

A question is a hit at rank r if the r-th retrieved chunk comes from the right page and contains the whole answer quote. If no chunk anywhere contains the whole quote, the answer was split by the chunker. That is a chunking failure no retriever can fix.

def run_benchmark(chunks: list[dict], items: list[dict], embed, k: int = 5,
                  use_path: bool = True) -> dict:
    texts = [f"{c['path']}\n\n{c['text']}" if use_path and c["path"] else c["text"]
             for c in chunks]
    chunk_vecs = embed(texts)
    question_vecs = embed([i["question"] for i in items])
 
    hits_at_1 = hits_at_k = 0
    reciprocal_ranks, split, misses = [], 0, []
 
    for item, qv in zip(items, question_vecs):
        quote = normalize(item["answer_quote"])
        holders = {j for j, c in enumerate(chunks)
                   if c["url"] == item["url"] and quote in normalize(c["text"])}
        if not holders:
            split += 1
            misses.append({"id": item["id"], "type": item["type"], "reason": "answer split"})
            reciprocal_ranks.append(0.0)
            continue
 
        ranked = sorted(range(len(chunks)), key=lambda j: cosine(qv, chunk_vecs[j]), reverse=True)
        rank = next((r for r, j in enumerate(ranked[:k], start=1) if j in holders), None)
        reciprocal_ranks.append(1 / rank if rank else 0.0)
        hits_at_1 += rank == 1
        hits_at_k += rank is not None
        if rank is None:
            misses.append({"id": item["id"], "type": item["type"], "reason": "not retrieved",
                           "top_chunk": chunks[ranked[0]]["text"][:200]})
 
    n = len(items)
    return {
        "chunks": len(chunks),
        "avg_chunk_chars": round(sum(len(c["text"]) for c in chunks) / max(len(chunks), 1)),
        "hit_at_1": hits_at_1 / n,
        f"hit_at_{k}": hits_at_k / n,
        "mrr": sum(reciprocal_ranks) / n,
        "split_rate": split / n,
        "misses": misses,
    }

Compare the strategies

corpus = json.load(open("corpus.json"))
items = json.load(open("test_set.json"))
assert not check_test_set(items, corpus)
 
heading = [c for page in corpus for c in heading_chunks(page)]
semantic = [c for page in corpus for c in semantic_chunks(page, embed)]
 
results = {
    "heading_with_path": run_benchmark(heading, items, embed, use_path=True),
    "heading_no_path": run_benchmark(heading, items, embed, use_path=False),
    "semantic": run_benchmark(semantic, items, embed),
}

Report every number, with its setup

Publish the full table, not just the winner. Fill it in from your own run:

Under the table, write down the setup: the date, the number of pages, the number of questions, the embedding model, k, max_chars and break_pct. Without that, nobody can repeat your test.

Also break the scores down by question type. A strategy can win overall and still lose badly on procedures or tables. That breakdown is often more useful than the overall score.

With a small test set, a gap of a few questions may be noise. If two strategies are close, run more questions before you decide.

Which errors appear in each approach?

Numbers tell you which strategy scored higher. The misses tell you why. Read every miss and tag it.

Common heading-aware errors

• Giant sections. A page with few headings becomes a few huge chunks. The answer is in there, but it is buried in unrelated text, so it ranks low.
• Tiny sections. A heading with one line below it makes a chunk with little meaning. The heading path helps here, which is why you test with and without it.
• Vague headings. Headings like "Overview" or "Notes" add little to the path.
• Fake headings. Some pages use bold text instead of real headings. The chunker cannot see those breaks.

Common semantic errors

• Broken procedures. Steps in a procedure can look different from each other, so the chunker splits between step 2 and step 3.
• Merged sections. Two short sections on related topics can end up in one chunk, which mixes their answers.
• Lost context. Without headings, a chunk that says "Set it to 300" may not say what "it" is.
• Setting sensitivity. Changing break_pct can change many chunk boundaries. Test more than one value.

Turn tags into a decision

Count the tags for each strategy. If most heading misses are "giant sections," try a smaller max_chars before you switch strategies. If most semantic misses are "broken procedures," that is a structural problem the setting may not fix.

Some teams find a mix works best: split by headings first, then use semantic splitting only inside sections that are still too long. Test that as a fourth row in your table if your error review points that way.

Hand off to your application

Once you pick a strategy, keep the benchmark. Save corpus.json, test_set.json and the results table together. When your docs change, fetch a new corpus, check that the answer quotes still appear and run the test again. A drop in hit rate tells you to look at the new pages before your users notice.

FAQs

How do headings define document sections?

In documentation, each heading names the topic of the text below it. Heading-aware chunking starts a new chunk at each heading, keeps the full heading path for context and splits long sections at paragraph breaks. It ignores # lines inside code blocks.

Which errors appear in each approach?

Heading-aware chunking can make chunks that are too large, too small or labeled with vague headings. Semantic chunking can split procedures, merge nearby sections and lose the context a heading would give. Reading and tagging every missed question shows which errors matter for your pages.

Which parts does MrScraper provide, and which parts must my application implement?

MrScraper is a developer-focused web data API. In this experiment, it discovers documentation URLs from a starting page and returns each page as Markdown with headings intact. Each response reports token usage.

Your application builds everything else: the saved corpus, both chunkers, the embeddings, the test set, retrieval, scoring and the error review. Your team also decides which pages it has permission to use.