Crawl Depth

Crawl depth is the number of link-hops a crawler follows from its starting URL. Depth 0 is the start page, depth 1 is every page it links to directly, depth 2 is everything those pages link to, and so on.

Set it too high on a large site and you spend your request budget on low-value pages. Set it too low and you miss content several clicks deep.

In SEO, crawl depth describes how far a page sits from the homepage and how that affects whether search engines find it. In data collection, it's a setting: the limit on how far a crawler travels from its start page.

How depth is counted

Depth counts link-hops, not URL path segments. A page at example.com/a/b/c/d/ linked directly from the homepage is at depth 1. A page at example.com/page/ reachable only through five successive clicks is at depth 5.

Depth 0:  /                          (start URL)
Depth 1:  /products/                 (linked from start)
Depth 2:  /products/category-a/      (linked from /products/)
Depth 3:  /products/category-a/item  (linked from category page)

Volume grows quickly because each level multiplies by the links per page. On a site averaging 30 internal links per page, depth 3 alone could hold up to 27,000 URLs before duplicates are removed. That's why an uncapped crawl of a large site is rarely what you want.

Crawl depth in SEO

For search engines, depth is a rough signal of importance. Pages many hops from the homepage tend to be crawled less often and receive less internal link equity.

• The 3-click guideline. An informal rule that important pages should be reachable within about three clicks of the homepage. It's a heuristic, not a ranking factor.
• Orphan pages. Pages with no internal links pointing to them. A link-following crawler never reaches them; only a sitemap or external link does.
• Crawl budget. Search engines spend a finite number of requests per site. Deep, low-value URLs (faceted navigation combinations, endless pagination, session-parameter duplicates) use it up.
• Site architecture. Reducing depth usually means better internal linking (hub pages, breadcrumbs, related-content modules), not restructuring URLs. Auditing it is a crawl in itself: see mapping and auditing entire websites.

Crawl depth in data collection

For a collection job, depth is a budget control. First decide whether it's the right control at all.

Don't use depth for pagination. Each ?page=N adds one hop, so a depth limit of 3 stops at page 3 of a 200-page listing. Handle pagination with an explicit page range or a "next" selector, and keep depth for real branching.

Use a URL list when you can. If you know which pages hold your data, a crawl list built from a sitemap, an API or a discovery pass beats open-ended exploration. A product job following every link from a category page will wander into navigation, footer, help-center and blog URLs; the same job given product URLs fetches only what it needs. See also mastering list crawling.

Combine depth with other limits. Depth alone doesn't bound a crawl. Pair it with a URL-pattern filter (stay within /products/), a page ceiling and a same-domain rule.

Setting a max depth in MrScraper

MrScraper's Scrape Sitemap endpoint runs the map agent, which follows a site's links from a start URL and returns the URLs it finds. It discovers pages only; it doesn't extract their content. This request caps discovery at two link-hops:

curl -X POST "https://api.mrscraper.com/?token=YOUR_API_TOKEN" \
  -H "x-api-token: YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://quotes.toscrape.com/",
    "agent": "map",
    "maxDepth": 2,
    "maxPages": 3,
    "limit": 1000
  }'

• maxDepth sets how many link-hops from the start URL the agent follows.
• maxPages caps how many pages it crawls to find links.
• limit caps how many URLs it returns. The response lists the discovered URLs with a count. Feed that list into an extraction job so you collect only the pages you need.

Choosing a depth

The shape of the site usually decides it:

• Depth 1–2. A known index or category page and its immediate children. Tight, predictable and cheap.
• Depth 3–5. General discovery when there's no sitemap and you need to find where content lives.
• Unlimited. Rarely right for a targeted job. Keep it for full audits of sites you own, with a hard page cap. A practical test: if you can't say what you expect to find at depth N that you wouldn't find at depth N−1, the limit is probably too high.