Skip to content
Build a Knowledge Graph From Web Pages With Source Provenance
Article

Build a Knowledge Graph From Web Pages With Source Provenance

Web

Turn permitted web pages into an evidence-backed knowledge graph. Define entities and relationships, extract triples with quotes, merge duplicates and audit every edge.

By MrScraper Team 13 min read

A knowledge graph stores facts as connections. "Acme Analytics integrates with Slack" becomes two nodes, Acme Analytics and Slack, joined by an INTEGRATES_WITH edge. Once facts are connected, you can ask questions a table makes hard, like "which tools connect to both Slack and Salesforce?"

A graph built from web pages has one big risk. After a few merges and updates, nobody remembers where an edge came from. Was it on the vendor's docs, or did the model guess it?

This tutorial builds a web data knowledge graph where every edge carries its evidence: the URL, the exact quote and the time the page was read. You will define a schema, extract triples, merge duplicate entities, load everything into a graph store and audit it.

MrScraper handles page discovery and extraction. Graph extraction rules, entity merging and storage are your own code.

The example: an integrations graph

Which software products integrate with which others, according to their own documentation?

This is a good first graph. Vendors list integrations on their own docs pages, the facts are concrete, and the result is easy to check. Use documentation sites your team owns or has permission to collect.

How do you define entities and relationships?

Start with a schema. It lists the types of things in your graph, the types of connections between them and which types each connection can join.

Keep the first schema small

Entity type Meaning Example
Product A software product or service Acme Analytics
Organization A company that makes products Acme Inc.
DataFormat A file or data format a product reads or writes CSV
Relationship From To Example
MADE_BY Product Organization Acme Analytics → Acme Inc.
INTEGRATES_WITH Product Product Acme Analytics → Slack
EXPORTS_TO Product DataFormat Acme Analytics → CSV

Three entity types and three relationships are enough to learn from. Add more only when you have a question that needs them.

Write the schema as code

The schema is also your validation rule. Any extracted triple that does not fit it gets rejected.

python
ENTITY_TYPES = {"Product", "Organization", "DataFormat"}
 
RELATIONS = {
    "MADE_BY": ("Product", "Organization"),
    "INTEGRATES_WITH": ("Product", "Product"),
    "EXPORTS_TO": ("Product", "DataFormat"),
}

Write a short definition for each relationship too. For example, "INTEGRATES_WITH means the page says the two products connect directly. Being listed on the same page is not enough." You will reuse that sentence in the extraction prompt.

How do you extract evidence-backed triples?

A triple is one fact: a subject, a relationship and an object. An evidence-backed triple also carries the sentence that states it.

Find the pages

The MrScraper Scrape Sitemap endpoint collects URLs from a starting page. Use includePatterns to keep integration and product pages.

python
import json
import os
import requests
from datetime import datetime, timezone
from urllib.parse import urlparse
 
MRSCRAPER_URL = "https://api.mrscraper.com/"
TOKEN = os.environ["MRSCRAPER_API_TOKEN"]
 
 
def call_mrscraper(body: dict, **params) -> dict:
    response = requests.post(
        MRSCRAPER_URL,
        params={"token": TOKEN, **params},
        headers={"x-api-token": TOKEN, "Content-Type": "application/json"},
        json={**body, "maxRetries": 2},
        timeout=330,
    )
    response.raise_for_status()
    return response.json()
 
 
def find_pages(start_url: str) -> list[str]:
    result = call_mrscraper({
        "url": start_url, "agent": "map", "maxDepth": 2, "maxPages": 10, "limit": 200,
        "includePatterns": [".*/(integrations|apps|connectors|export).*"],
    })
    site = urlparse(start_url).netloc
    return [u for u in (result.get("data") or {}).get("urls", []) if urlparse(u).netloc == site]

Extract triples with a schema

The MrScraper Scrape JSON endpoint takes a prompt and a JSON schema and returns structured data. It also returns the page html by default, which you will use to check each quote.

python
TRIPLE_SCHEMA = {
    "type": "object",
    "properties": {
        "triples": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "subject": {"type": "string"},
                    "subject_type": {"type": "string", "enum": sorted(ENTITY_TYPES)},
                    "predicate": {"type": "string", "enum": sorted(RELATIONS)},
                    "object": {"type": "string"},
                    "object_type": {"type": "string", "enum": sorted(ENTITY_TYPES)},
                    "quote": {"type": "string"},
                },
            },
        }
    },
}
 
PROMPT = (
    "List facts from this page that match these relationships:\n"
    "- MADE_BY: a product and the company that makes it.\n"
    "- INTEGRATES_WITH: the page says two products connect directly. "
    "Being listed on the same page is not enough.\n"
    "- EXPORTS_TO: a product can export data in a file or data format.\n"
    "Copy the exact sentence that states each fact into 'quote'.\n"
    "Json Schema:\n" + json.dumps(TRIPLE_SCHEMA)
)
 
 
def extract_triples(url: str) -> dict:
    return call_mrscraper({"url": url, "prompt": PROMPT, "agent": "general"})

Keep only triples that pass every check

Each triple must fit the schema, and its quote must appear on the page. This uses a small HTML-to-text helper that keeps inline tags like <b> from splitting a sentence.

python
from html.parser import HTMLParser
 
BLOCK_TAGS = {"p", "div", "li", "br", "tr", "td", "th", "h1", "h2", "h3", "h4", "section"}
 
 
class TextOnly(HTMLParser):
    def __init__(self):
        super().__init__()
        self.parts, self.skip = [], False
 
    def handle_starttag(self, tag, attrs):
        if tag in ("script", "style"):
            self.skip = True
        elif tag in BLOCK_TAGS:
            self.parts.append(" ")
 
    def handle_endtag(self, tag):
        if tag in ("script", "style"):
            self.skip = False
        elif tag in BLOCK_TAGS:
            self.parts.append(" ")
 
    def handle_data(self, data):
        if not self.skip:
            self.parts.append(data)
 
 
def normalize(text: str) -> str:
    return " ".join((text or "").split()).lower()
 
 
def page_text(html: str) -> str:
    parser = TextOnly()
    parser.feed(html or "")
    return normalize("".join(parser.parts))
 
 
def check_triple(t: dict, text: str) -> str | None:
    expected = RELATIONS.get(t.get("predicate"))
    if expected is None:
        return "unknown relationship"
    if (t.get("subject_type"), t.get("object_type")) != expected:
        return "types do not fit the relationship"
    if not t.get("quote") or normalize(t["quote"]) not in text:
        return "quote not found on page"
    if normalize(t["subject"]) not in normalize(t["quote"]) or normalize(t["object"]) not in normalize(t["quote"]):
        return "quote does not name both entities"
    return None
 
 
def collect(urls: list[str]) -> tuple[list[dict], list[dict]]:
    kept, rejected = [], []
    for url in urls:
        result = extract_triples(url)
        text = page_text(result.get("html", ""))
        retrieved_at = datetime.now(timezone.utc).isoformat()
        for t in (result.get("data") or {}).get("triples", []):
            reason = check_triple(t, text)
            record = {**t, "url": url, "retrieved_at": retrieved_at}
            if reason:
                rejected.append({**record, "reason": reason})
            else:
                kept.append(record)
    return kept, rejected

The last check matters more than it looks. A real quote that does not name both entities usually means the model joined two facts from different sentences. It also rejects sentences where the subject is only implied, like "You can export reports to CSV" on a product's own page. That is the cost of being strict. Keep the rejected list, review it and decide whether a rule like "on a product's own page, the product is the subject" is safe for your sources.

How do you deduplicate entities?

The same thing appears under many names. "Acme Analytics," "Acme analytics" and "Acme Analytics™" are one product. "Acme Inc." and "Acme, Inc." are one company. If you do not merge them, the graph splits one fact into many disconnected pieces.

Step 1: Normalize names into a key

python
import re
 
SUFFIXES = r"\b(inc|llc|ltd|corp|corporation|co|gmbh)\b\.?"
 
 
def name_key(name: str) -> str:
    key = name.lower()
    key = re.sub(r"[™®©]", "", key)
    key = re.sub(SUFFIXES, "", key)
    key = re.sub(r"[^a-z0-9]+", " ", key)
    return " ".join(key.split())

Entities of different types never merge. "Notion" the product and "Notion Labs" the company stay separate.

Step 2: Apply an alias list

Some names cannot be matched by rules. "Google Sheets" and "Sheets" may be the same product. Keep a short alias list in config, and have a person approve each entry.

python
ALIASES = {("Product", "sheets"): "google sheets", ("Product", "ms teams"): "microsoft teams"}
 
 
def entity_id(name: str, etype: str) -> str:
    key = name_key(name)
    key = ALIASES.get((etype, key), key)
    return f"{etype}:{key}"

Step 3: Flag near matches for review

Names that are close but not equal, like "Hubspot CRM" and "HubSpot," might be the same thing. Flag them instead of merging them. A wrong merge is worse than a missed one, because it silently connects facts that do not belong together.

python
from difflib import SequenceMatcher
from itertools import combinations
 
 
def near_matches(ids: set[str], threshold: float = 0.85) -> list[tuple[str, str, float]]:
    by_type = {}
    for i in ids:
        etype, key = i.split(":", 1)
        by_type.setdefault(etype, []).append(key)
    pairs = []
    for etype, keys in by_type.items():
        for a, b in combinations(sorted(keys), 2):
            ratio = SequenceMatcher(None, a, b).ratio()
            if ratio >= threshold or a in b.split() or b in a.split():
                pairs.append((f"{etype}:{a}", f"{etype}:{b}", round(ratio, 2)))
    return pairs

When a reviewer confirms a pair, add it to the alias list and reload. Every merge is then written down and can be undone.

Load the graph store

A graph store needs three things: nodes, edges and the evidence behind each edge. This example uses SQLite, so it runs anywhere. You can export the same tables into a dedicated graph database later.

python
import sqlite3
 
db = sqlite3.connect("graph.db")
db.executescript("""
CREATE TABLE IF NOT EXISTS nodes (id TEXT PRIMARY KEY, type TEXT, name TEXT);
CREATE TABLE IF NOT EXISTS edges (
    source TEXT, predicate TEXT, target TEXT,
    PRIMARY KEY (source, predicate, target)
);
CREATE TABLE IF NOT EXISTS evidence (
    source TEXT, predicate TEXT, target TEXT,
    url TEXT, quote TEXT, retrieved_at TEXT,
    PRIMARY KEY (source, predicate, target, url, quote)
);
""")
 
 
def load(triples: list[dict]) -> None:
    for t in triples:
        s = entity_id(t["subject"], t["subject_type"])
        o = entity_id(t["object"], t["object_type"])
        db.execute("INSERT OR IGNORE INTO nodes VALUES (?, ?, ?)", (s, t["subject_type"], t["subject"]))
        db.execute("INSERT OR IGNORE INTO nodes VALUES (?, ?, ?)", (o, t["object_type"], t["object"]))
        db.execute("INSERT OR IGNORE INTO edges VALUES (?, ?, ?)", (s, t["predicate"], o))
        db.execute("INSERT OR REPLACE INTO evidence VALUES (?, ?, ?, ?, ?, ?)",
                   (s, t["predicate"], o, t["url"], t["quote"], t["retrieved_at"]))
    db.commit()

Notice that an edge can have many evidence rows. If three pages say the same thing, the edge appears once, with three sources behind it.

How do you query and audit the graph?

Query the graph with ordinary SQL joins and recursive queries. Audit it by checking that every edge has evidence and by flagging edges that rest on weak or old sources.

Ask questions

Which products integrate with both Slack and Salesforce?

python
both = db.execute("""
    SELECT source FROM edges WHERE predicate = 'INTEGRATES_WITH' AND target = 'Product:slack'
    INTERSECT
    SELECT source FROM edges WHERE predicate = 'INTEGRATES_WITH' AND target = 'Product:salesforce'
""").fetchall()

Which products can reach Slack through one or two integration hops?

python
reach = db.execute("""
    WITH RECURSIVE path(node, hops) AS (
        SELECT source, 1 FROM edges WHERE predicate = 'INTEGRATES_WITH' AND target = 'Product:slack'
        UNION
        SELECT e.source, p.hops + 1 FROM edges e JOIN path p ON e.target = p.node
        WHERE e.predicate = 'INTEGRATES_WITH' AND p.hops < 2
    )
    SELECT node, MIN(hops) FROM path GROUP BY node
""").fetchall()

Show the evidence with every answer

Any answer from the graph should come with its sources. This function returns each edge's quotes and URLs.

python
def explain(source: str, predicate: str, target: str) -> list[dict]:
    rows = db.execute(
        "SELECT url, quote, retrieved_at FROM evidence WHERE source=? AND predicate=? AND target=?",
        (source, predicate, target),
    ).fetchall()
    return [{"url": u, "quote": q, "retrieved_at": r} for u, q, r in rows]

Run audit checks

Run these after every load:

python
def audit(max_age_days: int = 90) -> dict:
    cutoff = datetime.now(timezone.utc).timestamp() - max_age_days * 86400
    return {
        "edges_without_evidence": db.execute("""
            SELECT e.* FROM edges e LEFT JOIN evidence v
            ON e.source = v.source AND e.predicate = v.predicate AND e.target = v.target
            WHERE v.url IS NULL""").fetchall(),
        "single_source_edges": db.execute("""
            SELECT source, predicate, target FROM evidence
            GROUP BY source, predicate, target HAVING COUNT(DISTINCT url) = 1""").fetchall(),
        "stale_edges": [
            row for row in db.execute(
                "SELECT source, predicate, target, MAX(retrieved_at) FROM evidence "
                "GROUP BY source, predicate, target").fetchall()
            if datetime.fromisoformat(row[3]).timestamp() < cutoff
        ],
    }
  • Edges without evidence should never exist. If one shows up, a bug let a fact in without a source.
  • Single-source edges are not wrong, but they are the first place to look when an answer seems off.
  • Stale edges come from pages you have not read in a while. Fetch those pages again before you trust them. When a page changes, delete its old evidence rows, extract it again and reload. Then remove any edge left with no evidence. The graph only keeps facts that some current page still states.

Hand off to your application

Your application now has three tables it can serve: nodes to search, edges to traverse and evidence to show. When an answer is shown to a user, show the evidence with it, using explain. The rejected-triples list and the near-match list go to whoever maintains the schema and aliases.

FAQs

How do you define entities and relationships?

List a few entity types and relationship types, and say which types each relationship can join. Write a one-sentence definition for each relationship, including what does not count. Use the same definitions in your extraction prompt and your validation code, so anything outside the schema is rejected.

How do you query and audit the graph?

Query with joins for direct questions and recursive queries for multi-hop paths. Return the evidence with every answer. After each load, check for edges with no evidence, edges backed by only one source and edges from pages you have not read recently.

Which parts does MrScraper provide, and which parts must my application implement?

MrScraper is a developer-focused web data API. In this tutorial, it discovers URLs on a site and extracts candidate triples from each page with a prompt and schema. It also returns the page HTML, which you use to check each quote.

Your application builds the rest. This includes the schema, the triple checks, entity keys, the alias list, near-match review, the graph store, queries and audits. Graph extraction rules and storage are your own code. Your team also decides which sources it has permission to use.

Summarize this post

Open it in your assistant of choice with the prompt ready to send.

Take a Taste of Easy Scraping!
@MrScraper_

Find more insights here

Use Prompt Caching for Repeated Analysis of Web Documents
Guide

Use Prompt Caching for Repeated Analysis of Web Documents

Use your LLM provider's prompt caching when you analyze many extracted web pages. Keep stable prompt…

Measure and Reduce LLM Token Use in Web Data Applications
Web

Measure and Reduce LLM Token Use in Web Data Applications

Measure how many LLM tokens HTML, Markdown and field-only JSON use for the same pages, check answer…

2026 Residential Proxy Pricing: Comparing Enterprise Scale vs. Pay-As-You-Go
Proxies

2026 Residential Proxy Pricing: Comparing Enterprise Scale vs. Pay-As-You-Go

Compare 2026 residential proxies pricing: enterprise volume ladders vs. pay-as-you-go. Analysis of c…

Your choices

Cookie preferences

Necessary cookies keep your selection. Optional categories are disabled until you switch them on.

Strictly necessary

Remembers your privacy selection and keeps the site working.

Always on