Every tool here runs in your browser, free, with no signup and no install, and each one states its limit next to it.
Convert any page to clean Markdown
Strip the menus, scripts and footer from a page and keep its content as Markdown or plain text, ready for an AI prompt or a RAG pipeline.
Limit: 1 page per conversion · up to 100,000 characters · 30 a minute
Open the toolBuild an XML sitemap for any site
Crawl a site, keep only its indexable pages and download a sitemap.xml ready to submit to Google Search Console.
Limit: Up to 500 pages per crawl · one host
Open the toolCheck whether a URL is crawlable
Test any URL as Googlebot, GPTBot, ClaudeBot or a custom crawler and see the exact robots.txt line that allows or blocks it.
Limit: Files up to 500 KiB · unlimited retests · 30 fetches a minute
Open the toolPull every link from a page or text
Get every link from a page or a pasted block of text, labelled internal, external, image, mailto or tel, with a CSV export.
Limit: 1 page, first 2 MB of HTML · 30 fetches a minute · text mode unlimited
Open the toolDecode any UA string
Read the browser, version, operating system, device and engine from a user agent string, and spot crawler and HTTP-library strings.
Limit: No limit · runs entirely in your browser
Open the toolFour common jobs, with the tools in the order you would actually use them.
Check what the site permits, then build the URL list, then see what one page looks like once cleaned up.
Rule out robots.txt first, then check what your client announces itself as. A 403 or 429 usually means the site detected automation, not that the page is missing.
Collect the pages that matter, then turn each one into clean Markdown that spends context-window tokens on content instead of markup.
Confirm nothing important is blocked, see which pages are indexable, and list where each page links out to.
Yes. No tool on this page asks for an account, an email address or a trial, and none of them is a demo of a paid version.
Each tool states its own limit next to it: pages per crawl, characters per conversion, requests per minute. The limits exist to keep the tools fast for everyone, and they are the same for every visitor.
Web scraping is not illegal in itself. Collecting publicly available data is generally permitted, but three things change the answer: how the data is used, whether the terms of service of the site allow automated access, and whether personal data is involved.
Checking robots.txt before a crawl is the baseline of responsible automated access, though it is a request from the site owner rather than a law. This is not legal advice; for a specific project, read the terms and ask a lawyer.
JavaScript rendering. These tools read the HTML a server returns, so content that JavaScript adds after the page loads is invisible to them.
Anti-bot systems. CAPTCHAs, browser fingerprinting and IP bans stop a free tool. Getting past them takes a real browser and rotating IP addresses.
Scale and schedules. Free tools solve single-page problems. Thousands of URLs, structured output and runs on a schedule need an API.
When you reach those, MrScraper's Web Scraper API handles the rendering, the proxies and the scale.
Web scraping is not illegal in itself, and collecting publicly available data is generally permitted. How the data is used, whether the terms of service of the site allow automated access, and whether personal data is involved all affect the answer. For a specific project, read the terms and take legal advice.
ChatGPT can fetch and summarise individual pages when browsing is enabled, but ChatGPT is not a scraper. ChatGPT cannot crawl many pages, get past anti-bot systems or return structured data at scale on a schedule. Those jobs need a dedicated scraping tool or API that fetches, renders and parses pages for you.
Google does not allow automated access to its search results without permission, and it actively blocks scrapers of those results. Google does publish official APIs for some of its data. Its own crawlers, meanwhile, follow the robots.txt rules that other sites set, which is the same courtesy it asks of everyone else.
Free tools that scrape Google search results exist, but they are unreliable because Google blocks automated queries aggressively. Results break without warning when a request is flagged or a CAPTCHA appears. For consistent results, a SERP API with managed proxies is the practical option, since it absorbs the blocking for you.
Every tool on this page is free, with no account, no email address and no trial. Each tool runs from your browser and lists its own limit on its page, such as 500 pages per crawl or 100,000 characters per conversion, so there is nothing hidden in the small print and no paid tier behind it.
Most web scraping projects need four things: a way to find URLs, a way to check crawl permission, a way to test selectors, and a way to extract clean content. A sitemap generator, a robots.txt tester, an XPath or CSS selector tester and a page-to-text converter cover all four jobs between them.
Nothing needs to be installed to use the tools on this page. Every tool runs in the browser you are reading this in, so there is no extension to add, no desktop application to download and no command-line package. A phone browser works too, although the larger results are easier to read on a desktop.
A crawler discovers and follows URLs across a site, while a scraper extracts specific data from pages it already has. Most real projects use both: crawl to find the pages, then scrape to pull the data from each one. The sitemap generator here is a small crawler, and the converters are small scrapers.
Sites block automated traffic with rate limits, browser fingerprinting and CAPTCHA challenges, and a free browser-based tool cannot get past those. A 403 or 429 response usually means the site detected automation. Handling blocks requires rendering the page in a real browser, pacing requests and rotating IP addresses.
Start with the robots.txt tester to check whether the site permits crawling, then use the sitemap generator to find the URLs. After that, run one page through the website-to-text converter to see what clean output looks like. That order mirrors how a real scraping project begins, from permission to discovery to extraction.
Starting a scraping project? Check permission, find the URLs, then look at one clean page.
