Crawl Depth

Crawl depth is the number of link-hops a crawler follows from its starting URL. Depth 0 is the seed page itself, depth 1 is every page it links to directly, depth 2 is everything those pages link to, and so on. Setting it too high on a large site burns request budget on low-value pages; setting it too low can miss the content you actually need several clicks deep.

The term carries slightly different weight depending on who's using it. In SEO, crawl depth describes how far a page sits from the homepage and how that distance affects whether search engines find and value it. In scraping, it's a configuration parameter, the cap you set on how far a crawler wanders from its seed before it stops.

How Depth Is Counted

Depth is measured in link-hops, not URL path segments. A page at example.com/a/b/c/d/ linked directly from the homepage is at depth 1, even though its URL is four levels deep. Conversely, a page at example.com/page/ that's only reachable through five successive clicks sits at depth 5.

Depth 0:  /                          (seed URL)
Depth 1:  /products/                 (linked from seed)
Depth 2:  /products/category-a/      (linked from /products/)
Depth 3:  /products/category-a/item  (linked from category page)

Because each level multiplies out by the average number of links per page, crawl volume grows fast. A site averaging 30 internal links per page reaches roughly 27,000 unique URLs by depth 3 before deduplication, which is why an uncapped crawl on a large site is rarely what anyone actually wants.

Crawl Depth in SEO

For search engines, depth is a rough proxy for importance. Pages buried many hops from the homepage get crawled less often and tend to accumulate less internal link equity.

• The 3-click guideline: a common (if informal) recommendation that important pages should be reachable within about three clicks of the homepage. It's a heuristic, not a ranking factor.
• Orphan pages: pages with no internal links pointing to them at all. They have no crawl depth in practical terms, because a link-following crawler will never reach them. They're only discoverable via sitemap or external link.
• Crawl budget: search engines allocate a finite number of requests per site. Deep, low-value pages (faceted-navigation URL combinations, endless pagination, session-parameter duplicates) consume that budget without returning anything useful.
• Site architecture: flattening depth usually means better internal linking (hub pages, breadcrumbs, related-content modules) rather than literally restructuring URLs. Auditing this is a crawl in itself: see mapping and auditing entire websites with web scraping.

Crawl Depth in Web Scraping

For a scraper, depth is a budget control. The key question is whether depth is even the right mechanism for the job.

Depth is the wrong tool for pagination. A paginated listing (?page=1, ?page=2, ?page=3…) increments depth by one on every page, so a depth cap of 3 stops you at page 3 of a 200-page listing. Pagination should be handled by an explicit page-range loop or a "next" selector, with depth reserved for genuine branching.

Prefer an explicit URL list where you can. If you already know which pages hold your data, a crawl list in web scraping, built from a sitemap, an API, or a discovery pass, beats depth-based exploration outright. A product scraper following every link from a category page will chase navigation, footer, help-center, and blog URLs; the same scraper given a list of product URLs fetches exactly what it needs. The same logic applies when mastering list crawling.

Combine max crawl depth with other limits. Depth alone doesn't bound a crawl usefully. In practice you want depth plus a URL-pattern filter (stay within /products/), a max-pages ceiling, and a same-domain restriction. Most frameworks expose all four:

Choosing a Depth

There's no universal right answer, but the shape of the target usually decides it:

• Depth 1–2: scraping a known index or category page and its immediate children. Tight, predictable, cheap.
• Depth 3–5: general site discovery when you don't have a sitemap and need to find where content lives.
• Unlimited: rarely appropriate for a targeted scrape. Reserve it for full-site audits on sites you own, paired with a hard page cap. The practical test: if you can't articulate what you expect to find at depth N that you wouldn't find at depth N−1, the cap is probably too high.