Web Scraping Glossary
Plain-English definitions for the scraping, proxy, and anti-bot terms you'll run into building a scraper.
23 terms and counting
A
-
AI Crawler
Learn the definition of an AI crawler, an automated bot operated by an AI company that visits websites to collect content for training large language models, powering real-time answers, or generating citations in AI search products.
Read more → -
Anti-Bot System
An anti-bot system is a layered defense mechanism analyzing IP reputation, fingerprints, and behavior to block automated scrapers.
Read more → -
Anti-Fingerprinting
Learn the definition of anti-fingerprinting, spoofing, or randomizing canvas, WebGL, fonts, and screen attributes so a browser doesn't produce the same identifiable hash.
Read more → -
ASN (Autonomous System Number)
An ASN identifies the network operator owning an IP block, allowing anti-bot systems to classify traffic as residential ISP or datacenter.
Read more →
B
-
Browser Fingerprinting
Browser fingerprinting builds a composite identity from hardware and rendering signals to detect scrapers, persisting even when IP addresses change.
Read more →
C
-
Crawl Depth
Learn what crawl depth means and how to set the max crawl depth for both SEO audits and web scraping jobs.
Read more → -
CSS Selector
A CSS selector targets HTML elements by class, ID, or attributes in web scraping. Compare CSS selector vs XPath performance and essential scraper syntax.
Read more →
D
-
Datacenter Proxy
Learn the definition of a datacenter proxy, an IP hosted in a commercial data center rather than an ISP, and how it compares to residential proxies for web scraping.
Read more →
H
-
Headless Browser
Learn the definition of a headless browser, a full browser engine that renders pages and runs JavaScript without a visible UI and why anti-bot systems can fingerprint it.
Read more → -
Headless Browser Detection
Learn how sites detect headless browsers, navigator.webdriver, empty plugin arrays, WebGL renderer strings, and the JavaScript probes that identify automation.
Read more → -
Honeypot Trap
Learn the definition of a honeypot trap, a link or form field hidden from human users that flags automated scrapers and bots the moment they interact with it.
Read more →
I
-
IP Reputation
IP reputation is a trust score assigned to IP addresses based on ASN type and abuse history, determining whether scrapers get challenged or allowed.
Read more → -
IP Rotation
IP rotation cycles requests through a large proxy pool to avoid rate limits and bans. Understand how rotating IP addresses work for web scraping.
Read more →
J
-
JavaScript Rendering
JavaScript rendering builds page content dynamically in the browser, requiring headless browsers to extract data from modern client-side apps.
Read more →
L
-
llms.txt
Learn the definition of llms.txt, a proposed markdown file format placed at a website's root to provide AI models and large language models with a curated summary of key content and documentation.
Read more →
P
-
Proxy Authentication
Proxy authentication verifies client access via user:password credentials or IP whitelisting before routing requests to prevent 407 unauthorized proxy errors.
Read more → -
Proxy Bandwidth
Proxy bandwidth is the data volume in GB metered by residential proxy providers. Learn how page weight and media assets drive proxy bandwidth cost.
Read more → -
Proxy Pool
A proxy pool is the set of IP addresses a scraper rotates across requests to distribute traffic and avoid blocks.
Read more →
S
-
Stealth Mode
Learn the definition of stealth mode in browser automation, the patches that hide navigator.webdriver and other automation signals, and what they can't fix.
Read more → -
Sticky Session (Session Persistence)
Technical definition of sticky sessions in web scraping: IP persistence mechanics, TTL specifications, and session ID proxy parameters.
Read more →
T
-
TLS Fingerprinting (JA3 / JA4)
Learn the definition of TLS fingerprinting, a security mechanism that analyzes initial handshake patterns to identify and block automated clients using standards like JA3 and JA4, regardless of IP rotation or headers.
Read more →
U
-
User Agent
Technical definition of User-Agent headers in web scraping: string format anatomy, modern UA reference list, Client Hints (Sec-CH-UA), and browser fingerprint alignment.
Read more →
W
-
Web Crawler
A web crawler systematically discovers and follows links across the web to build an index, differentiating it from a web scraper.
Read more →
No terms match that search yet.
More terms are added every week.
Build at enterprise scale. Scrape at startup speed
High-performance data extraction you can deploy in minutes - reliable, compliant, and built for growth.