Robots.txt Tester: Check Whether a URL Is Crawlable
A free replacement for the retired Search Console robots.txt tester. Enter a URL, pick a crawler, and see whether it is allowed and exactly which line decides it.
How to test a URL against robots.txt
- 1
Enter the URL you want to check
The tester reads the robots.txt at the root of that host. Subdomains have their own file, so blog.example.com is tested separately.
- 2
Pick the crawler
Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot or any custom token. Two crawlers can get different answers from the same file.
- 3
Read the verdict and the line
Allowed or blocked, the rule that decided it, the group it came from, and any Crawl-delay or Sitemap lines. Edit the file to test a fix.
Which robots.txt rule wins
Most robots.txt mistakes come from assuming the first matching line decides. RFC 9309, the Robots Exclusion Protocol standard, says otherwise: the most specific rule does.
User-agent: *
Disallow: /shop/
Allow: /shop/sale
Disallow: /*.pdf$-
/shop/cartblocked Only Disallow: /shop/ matches. -
/shop/sale/shoesallowed Allow: /shop/sale (10 characters) beats Disallow: /shop/ (6). -
/guides/intro.pdfblocked The asterisk matches /guides/intro and $ anchors .pdf to the end. -
/guides/intro.pdf?v=2allowed The query string means the URL no longer ends in .pdf.
-
The longest match wins
When several rules match a path, the one with the longest pattern decides. Position in the file is irrelevant.
-
Allow wins a tie
If an Allow and a Disallow rule match with patterns of equal length, the URL is allowed.
-
One group per crawler
A crawler reads only the group that names its own token, and falls back to User-agent: * only when no group names it.
-
Trailing slashes matter
Disallow: /folder blocks /folder, /folder/ and /folder-archive. Disallow: /folder/ blocks only what sits inside the folder.
AI crawler user-agent tokens
Each AI company crawls under its own token, and some tokens only control how content is used rather than who fetches it.
| Token | Operator | What it does | robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Collects content for training foundation models. | Obeys |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search results. | Obeys |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks for it. | Not applied to user-initiated fetches |
| ClaudeBot | Anthropic | Collects content that may be used for model training. | Obeys, including Crawl-delay |
| Claude-SearchBot | Anthropic | Improves the quality of Claude search results. | Obeys |
| Claude-User | Anthropic | Fetches a page when a Claude user asks for it. | Obeys |
| PerplexityBot | Perplexity | Indexes and links sites in Perplexity answers; not used for model training. | Obeys |
| Perplexity-User | Perplexity | Visits pages a Perplexity user asks about. | Generally ignores |
| Google-Extended | Control token, not a crawler: opts content out of Gemini training without affecting Search. | Read by existing Google crawlers | |
| Applebot-Extended | Apple | Control token, not a crawler: opts content out of Apple foundation model training. | Read by Applebot |
| CCBot | Common Crawl | Builds the open Common Crawl corpus that many models train on. | Obeys |
What people use a robots.txt checker for
-
Diagnose "Blocked by robots.txt"
Paste the URL from the Search Console report and the tester names the rule responsible.
-
Check an edit before it ships
Change the file in the editor and retest, so a stray Disallow never reaches production.
-
Audit AI crawler access
Switch between GPTBot, ClaudeBot and PerplexityBot to see which AI crawlers a site admits.
-
Check a site before scraping it
Confirm which paths a site asks automated clients to avoid before a single request is sent.
When this free tool is not enough
-
One live file at a time. The tester fetches robots.txt from the root of the host you enter, up to 500 KiB, the size RFC 9309 requires crawlers to read.
-
Crawl rules, not index rules. A noindex tag, a canonical or a login wall can still keep a crawlable URL out of results. The tester reads only robots.txt.
-
No bulk checks or monitoring. URLs are tested one by one, and nothing watches the file for changes. Fair use is 30 fetches a minute.
Checking crawl rules across thousands of URLs, or scraping the pages a site allows, is what MrScraper's Web Scraper API handles.
How to check robots.txt with code
Python's built-in urllib.robotparser applies rules in file order rather than by longest match, so it can disagree with Google on files with overlapping rules. The robots-parser package for Node follows the longest-match rule.
from urllib import robotparser
parser = robotparser.RobotFileParser("https://example.com/robots.txt")
parser.read()
print(parser.can_fetch("GPTBot", "https://example.com/private/page"))
print(parser.crawl_delay("Bingbot"))import robotsParser from 'robots-parser'
const url = 'https://example.com/robots.txt'
const robots = robotsParser(url, await (await fetch(url)).text())
console.log(robots.isAllowed('https://example.com/private/page', 'GPTBot'))
console.log(robots.getMatchingLineNumber('https://example.com/private/page', 'GPTBot'))Robots.txt tester FAQ
Is robots.txt legal?
Robots.txt is a voluntary standard, not a law, and it does not technically prevent access to anything. The file tells crawlers which paths a site prefers they avoid. Ignoring it is not automatically illegal, though it may breach the terms of service of the site, and how the collected data is used matters too.
Does robots.txt still work?
Robots.txt still works for every major search engine and AI crawler that chooses to follow it. Google, Bing, GPTBot, ClaudeBot and PerplexityBot all fetch the file and respect its rules. The file does not block anything at the server level, so a badly behaved crawler can ignore it, and server-side blocking is the only enforcement.
How do I fix "Blocked by robots.txt"?
A "Blocked by robots.txt" error is fixed by removing or narrowing the rule that matches the blocked URL. Open the file, find the Disallow line the tester names, then delete it or add a more specific Allow rule. Request re-indexing in Search Console afterwards, and expect the change to take a few days to show.
How do I view a robots.txt file?
Any robots.txt file can be viewed by adding /robots.txt to the domain and opening it in a browser, for example example.com/robots.txt. The file is always public and always sits at the root of the host. A 404 response means the site has no robots.txt, and all crawling is permitted by default.
Which robots.txt rule wins when two rules match?
The rule with the longest matching pattern wins when two robots.txt rules match the same URL. Allow: /shop/sale beats Disallow: /shop/ for /shop/sale/shoes because its pattern is longer. When an Allow and a Disallow rule are equally long, Allow takes precedence. The order of the lines in the file never decides the outcome.
How do I block AI crawlers in robots.txt?
AI crawlers are blocked by adding a user-agent group for each one, such as User-agent: GPTBot followed by Disallow: /. Each company uses its own token: GPTBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity and CCBot for Common Crawl. Google-Extended and Applebot-Extended are control tokens that opt content out of model training.
What is the difference between robots.txt and noindex?
Robots.txt stops a crawler from fetching a page, while a noindex tag lets the crawler fetch the page but tells it not to index it. The two conflict in a way that surprises people: a page blocked in robots.txt is never fetched, so its noindex tag is never read, and the URL can still appear in results.
Does robots.txt block a page from Google search?
Robots.txt does not reliably keep a page out of Google search, because it prevents crawling rather than indexing. When other sites link to a blocked URL, Google can still list it without a description. A noindex tag on a crawlable page is the dependable way to keep a URL out of search results.
What replaced the Google Search Console robots.txt tester?
Google retired the robots.txt Tester in Search Console and replaced it with a robots.txt report. The report lists the files Google fetched for a property, when it fetched them and any parsing errors, but it cannot test an individual URL against the rules. Third-party testers such as this one fill that gap.
Do wildcards work in robots.txt?
Wildcards work in robots.txt for Google, Bing and the major AI crawlers, and RFC 9309 standardised them in 2022. An asterisk matches any sequence of characters, and a dollar sign anchors the pattern to the end of the URL. Disallow: /*.pdf$ blocks every URL ending in .pdf but not one ending in .pdf?v=2.
Should I check robots.txt before scraping a site?
Checking robots.txt before scraping a site is the baseline of responsible automated access. The file shows which paths the owner asks automated clients to avoid and sometimes how fast to go. The file is not a complete legal position, though, because the terms of service of the site and data protection law apply independently.
What does Crawl-delay do?
Crawl-delay asks a crawler to wait a set number of seconds between requests to the site. Google does not support the directive and ignores it, managing its crawl rate automatically instead. Bing, ClaudeBot and several other crawlers do honour it, so a Crawl-delay line still slows those clients down.