Is Web Scraping Legal?

The question "is web scraping legal?" gets asked millions of times a year, and the most accurate answer is also the most frustrating one: it depends. Depends on what you're scraping, who you're scraping it from, what you plan to do with the data, which country's laws apply, and whether personal data is involved. There is no single global answer. What's clearly permitted in one jurisdiction and context can be a legal gray area or outright prohibited in another.

Web scraping legality sits at the intersection of several distinct legal frameworks — computer access law, copyright law, contract law through website Terms of Service, and data protection regulations — each of which applies differently based on the specifics of a given scraping operation. Understanding which frameworks apply to your situation is the first step toward operating responsibly. This article breaks down each framework honestly, covers the key legal cases that have shaped the landscape, and gives you a practical framework for evaluating the legality of a specific scraping project.

Note: This article is informational and educational. It is not legal advice and should not be treated as such. For specific legal questions about your scraping activities, consult a qualified attorney familiar with the relevant jurisdictions and applicable law.

The Short Answer: It Depends on Four Factors

Web scraping exists in a complex legal environment where the same technical activity — making automated HTTP requests to websites and collecting the returned data — can be legal, legally ambiguous, or clearly illegal depending on four factors:

1. What is being scraped. Publicly accessible information about businesses, organizations, or factual data carries different legal risk than personal data about identifiable individuals, copyrighted content, or data that sits behind authentication. The data type determines which legal frameworks engage.

2. Who owns it and on what terms. Websites' Terms of Service are contracts that users agree to and that typically restrict or prohibit automated scraping. ToS violations are generally a civil contract matter rather than a criminal one, but platforms have pursued legal action on ToS grounds. Additionally, database protection laws (particularly in the EU) can extend protection to compilations of data even when the underlying facts are unprotected.

3. How the data is used. Scraping factual information for personal research or internal analysis raises different issues than republishing scraped content commercially, using it to compete with the source platform, or aggregating personal data for profiling.

4. Where you and the target are located. The Computer Fraud and Abuse Act (CFAA) applies to activity in the United States. GDPR applies to personal data of EU residents regardless of where the scraper is located. Different countries have different computer access, copyright, and data protection laws that may apply simultaneously.

The Four Legal Frameworks That Apply to Web Scraping

1. The Computer Fraud and Abuse Act (CFAA) — United States

The CFAA, enacted in 1986 and amended multiple times, prohibits "unauthorized access" to computer systems. It's the primary statute US courts have applied to web scraping cases, and its interpretation has been contested significantly.

The central question in scraping cases under the CFAA is what "without authorization" means when applied to publicly accessible websites. Two interpretations have competed: the broader view (that violating a website's Terms of Service constitutes unauthorized access) and the narrower view (that authorization extends to anyone who can access the information without a password or technical barrier).

The Supreme Court's 2021 decision in Van Buren v. United States significantly clarified the CFAA's scope. The Court held that the CFAA applies to situations where someone accesses computer systems they're not entitled to access at all — not merely where they access authorized information in ways that violate a platform's policies. This decision narrowed the CFAA's reach and supports the position that scraping publicly accessible web pages (those without authentication requirements) is generally not a CFAA violation, even if the target site's Terms of Service prohibit scraping.

However, "generally not a CFAA violation" is not the same as "clearly legal." Circumventing technical barriers, accessing gated content, or using credentials to access data beyond authorized scope all remain potential CFAA exposure. And terms like "unauthorized access" are still litigated.

2. Copyright Law

Web pages contain copyrighted content — written articles, photographs, videos, proprietary data compilations. Copyright protects creative expression; it doesn't protect facts, ideas, or information in isolation.

Scraping facts, data points, prices, and similar uncopyrightable information is generally not a copyright violation. Reproducing full articles, images, or other creative works — even by automated means — potentially is. The key distinction is between scraping data (extracting factual information from a page) and reproducing content (copying the page's creative expression).

Compilation copyright is a separate consideration. A database that requires substantial selection, arrangement, or investment to compile may be protected as a compilation even if the individual data points are facts. In the EU, the Database Directive provides specific protection for databases where there was substantial investment in obtaining, verifying, or presenting the contents — protection that can cover data compilations beyond what copyright law alone would cover.

3. Terms of Service (ToS) — Contract Law

Website Terms of Service are contracts. When a ToS prohibits automated access or scraping, violating that prohibition is a breach of contract. Whether any specific user actually agreed to the ToS — by clicking "I agree," by using the site after the ToS is presented, or simply by using a site that displays the ToS — is sometimes contested.

The legal consequences of ToS violations are typically civil: the platform can terminate your access, sue for damages in contract, or seek an injunction. Whether a ToS violation alone creates criminal liability (via CFAA) has been specifically tested and, post-Van Buren, courts are less likely to treat ToS violations as criminal "unauthorized access" on publicly accessible sites.

Importantly, ToS provisions can restrict use in ways that are independently valid as contracts even where the underlying scraping activity might otherwise be permissible. A platform's ToS prohibiting commercial use of their data can be enforceable through contract law even if the underlying scraping of public data is not a CFAA violation.

4. Data Protection Regulations — GDPR, CCPA, and Similar

Personal data protection is where the clearest and most consistently enforced legal obligations apply to web scraping.

GDPR (EU) requires a lawful basis for collecting, processing, or storing personal data — data that relates to an identifiable natural person. This applies regardless of where the scraper is located if the data concerns EU residents. Collecting names, contact information, employment history, or any other personally identifiable information through scraping without a lawful basis is a GDPR violation, potentially subject to significant fines.

CCPA (California) requires businesses that collect personal data about California residents to meet disclosure, opt-out, and other compliance obligations. Large-scale scraping of personal data about Californians for commercial purposes triggers these requirements.

The critical insight is that publicly visible personal data — a person's name and employer on a LinkedIn profile, a reviewer's name and photo on a Yelp page — is still personal data under GDPR and similar frameworks. Public visibility doesn't remove data protection obligations. Scraping individual people's data at scale almost certainly requires legal basis analysis, regardless of where that data appears.

Key Legal Cases That Have Shaped the Landscape

hiQ Labs v. LinkedIn is the most significant US case on scraping publicly accessible data. LinkedIn sued hiQ (a people analytics company) after it sent a cease-and-desist letter prohibiting hiQ from scraping LinkedIn's public profile data. The 9th Circuit Court of Appeals initially ruled in 2019 that hiQ could continue scraping LinkedIn's publicly accessible data, finding that such scraping likely didn't violate the CFAA because the data was publicly available without authentication.

After the Supreme Court's 2021 Van Buren decision, the 9th Circuit reconsidered the case and again, in 2022, ruled largely in hiQ's favor on the CFAA question — holding that accessing publicly available data without circumventing any technical barriers is not "without authorization" under the CFAA. However, the court remanded some issues, and the case ultimately didn't resolve definitively for hiQ as a company, which ceased operations separately. The hiQ rulings remain significant precedent for the position that CFAA doesn't prohibit scraping publicly accessible data.

Van Buren v. United States (2021) is not a scraping case but is foundational to scraping law. The Supreme Court held that the CFAA's "exceeds authorized access" language applies to situations where someone accesses information on a computer system they're not entitled to access — not to situations where someone accesses information they're authorized to access but for improper purposes. This narrowing of the CFAA reduces the risk that ToS violations alone constitute criminal "unauthorized access" for scraping publicly available content.

Ryanair v. PR Aviation (EU, 2015) and related European cases address database rights. The European Court of Justice held in Ryanair that while database rights can protect compilations, Terms of Service restricting access can still be enforceable under contract law. EU database protection adds a layer of legal consideration for scraping structured data compilations that doesn't exist in the same form in US law.

Step-by-Step: Evaluating the Legality of Your Scraping Project

Step 1: Identify the Type of Data You're Collecting

Publicly accessible factual information about businesses, products, or prices: Generally lower legal risk. Facts aren't copyrightable, and public availability reduces CFAA exposure under current US precedent.

Personal data about individuals: Immediately engages GDPR, CCPA, and similar frameworks. Requires legal basis analysis. The publicly visible status of the data doesn't reduce this obligation.

Copyrighted creative content (articles, images, videos, unique expression): Copyright law applies. Collecting for personal reference differs from reproduction and redistribution.

Data behind authentication: Accessing data through credentials in ways that exceed authorized scope is higher CFAA risk. Circumventing authentication mechanisms is clearly problematic.

Step 2: Review the Target's Terms of Service

Most websites prohibit scraping in their Terms of Service. Read the relevant sections and understand:

Does the ToS prohibit automated access or scraping?
Does the ToS restrict commercial use of data?
What does the ToS say about enforcement?

ToS violations create civil liability risk regardless of whether they create criminal risk. Platforms that have pursued litigation against scrapers often do so on ToS grounds combined with other claims. Evaluating that risk — for your specific use case and scale — requires judgment and often legal advice.

Step 3: Check robots.txt

The robots.txt file at the root of a website specifies which automated crawlers should avoid which pages. It's not legally binding in most jurisdictions, but it's ethically significant and has been relevant in some legal arguments about whether access was "authorized." Respecting robots.txt is a standard practice in responsible scraping.

According to the W3C Robots Exclusion Protocol specification, robots.txt is a convention for communicating crawling preferences to automated agents. Courts have varied on how much legal weight to give robots.txt compliance or non-compliance.

Step 4: Analyze Your Data Use

The same scraping activity carries different legal implications depending on what you do with the data:

Internal analytics and research: Generally lower risk than commercial redistribution
Building competing products using a platform's own data: Higher risk (contractual, potentially copyright if compilations are protected)
Republishing scraped content: Copyright risk if creative expression is involved
Processing personal data for profiling or targeting: Engages data protection obligations regardless of use

Step 5: Consider Your Jurisdiction and the Target's Jurisdiction

GDPR applies to data of EU residents regardless of where you're located. US law applies to systems based in the US. Some countries have broader or narrower computer access laws. Cross-border scraping may trigger multiple legal frameworks simultaneously.

Step 6: Consult Legal Counsel for Commercial Applications

For individual research or hobby projects at minimal scale, the practical risk of most public data scraping is low. For commercial applications — particularly at scale, involving personal data, competing with the source platform, or involving jurisdictions with strong database protection or data privacy laws — the analysis is complex enough to warrant legal advice specific to your situation.

Common Challenges and Gray Areas

The public/private data distinction is blurrier than it appears. "Public" means accessible without authentication — but platforms can argue that scraping at scale exceeds the implicit license granted by public access. What a single user can view isn't necessarily what a service can systematically collect. Courts have generally been skeptical of this distinction for publicly accessible data under the CFAA, but it remains contested in other legal frameworks and jurisdictions.

ToS agreements are enforceable contracts regardless of CFAA outcomes. Even where courts hold that scraping public data doesn't violate the CFAA criminally, scraping in violation of a ToS remains a civil contract breach. Platforms can pursue damages and injunctions under contract law independent of criminal computer access claims. The narrowed CFAA interpretation from Van Buren and hiQ doesn't make ToS violations legally risk-free — it just reduces the criminal exposure specifically.

Personal data protection is the clearest and most enforced legal constraint. Unlike the ambiguities in CFAA and ToS interpretation, GDPR enforcement against unauthorized collection of personal data is well-established and active. Data protection authorities have pursued enforcement actions involving scraping, including cases involving LinkedIn data and other platforms. If personal data of EU residents is involved, treat compliance as mandatory, not optional.

AI training data scraping has opened new legal questions. The use of scraped web content to train large language models and other AI systems has generated significant new litigation — from publishers, authors, visual artists, and others — raising copyright questions that aren't yet resolved in courts. This is an actively evolving area of law with significant uncertainty.

Platforms update their legal position independently of court decisions. Even where courts have ruled that certain scraping activity is permissible, target platforms can implement technical countermeasures, update their ToS, or pursue new legal theories. Legal permissibility at a point in time doesn't mean sustained unchallenged access.

Conclusion

The honest answer to "is web scraping legal?" is that it's a question with multiple layers, not a binary answer. Scraping publicly accessible factual data for personal research or non-commercial purposes, while respecting robots.txt and not circumventing authentication, sits in a lower-risk zone that US courts have generally not treated as criminally prohibited under the CFAA following Van Buren and hiQ. That's a significant data point.

But "lower risk" and "clearly legal" are different things. Terms of Service violations create civil liability risk. Personal data protection laws create compliance obligations that public visibility doesn't remove. Copyright law applies to creative content. Database rights (especially in the EU) can protect compilations. Cross-border situations engage multiple frameworks simultaneously.

The practical implication is that context determines the answer, and commercial applications at scale — particularly those involving personal data, EU data subjects, or competing with the source platform — warrant specific legal analysis, not reliance on general guidance. For those applications, counsel familiar with computer, copyright, contract, and data protection law in the relevant jurisdictions is the appropriate resource.

What We Learned

Web scraping legality depends on four factors: what's scraped, who owns it and on what terms, how it's used, and applicable jurisdiction — none of these alone determines the answer.
The CFAA's scope has been narrowed by Van Buren (2021) and hiQ Labs (2022): Current US precedent suggests that scraping publicly accessible data without circumventing authentication is generally not criminal "unauthorized access" under the CFAA.
ToS violations are civil matters but remain legally risky: Scraping in violation of Terms of Service creates civil liability exposure through contract law independent of CFAA criminal risk — platforms have pursued and won injunctions and damages on these grounds.
Personal data carries the clearest and most consistently enforced legal obligations: GDPR, CCPA, and similar regulations apply to personal data regardless of its public visibility, and enforcement is well-established.
Copyright law protects creative expression, not facts: Scraping factual data is generally not copyright infringement; reproducing articles, images, or copyrighted creative works is.
Commercial applications at scale require legal analysis, not general guidance: The combination of ToS exposure, data protection compliance, copyright considerations, and jurisdictional complexity makes legal counsel the appropriate resource for commercial scraping operations.

FAQ

Is web scraping illegal?

Web scraping is not categorically illegal — it depends on what is scraped, how it's used, and which legal frameworks apply. Scraping publicly accessible factual information for research or analysis occupies legally lower-risk territory under US law (following Van Buren and hiQ precedent). However, scraping personal data triggers data protection obligations (GDPR, CCPA), scraping in violation of Terms of Service creates civil liability, and reproducing copyrighted creative content raises copyright issues. Context determines the legal analysis.
Can websites legally stop you from scraping their data?

Yes, through multiple mechanisms. Terms of Service that prohibit scraping are enforceable contracts — platforms can pursue civil action for damages or injunctions against scrapers who violate their ToS. Technical measures that block scrapers are lawful, and circumventing them may create legal exposure. Platforms with database rights (particularly in the EU) can assert protection over data compilations. Whether scraping itself is criminal or civilly actionable depends on specifics, but platforms have meaningful legal tools to protect their data.
Does respecting robots.txt make web scraping legal?

Respecting robots.txt is considered ethically important in the scraping community and reflects responsible data collection practices. However, it doesn't by itself make scraping legally permitted — it's one factor, not a safe harbor. A website with robots.txt directives still has Terms of Service, copyright protections, and data protection obligations that apply independently. Conversely, not respecting robots.txt can be cited as evidence of bad faith in litigation even if it's not independently illegal in most jurisdictions.
Is scraping personal data legal under GDPR?

Scraping personal data about EU residents requires a lawful basis under GDPR regardless of the data's public visibility. The fact that information appears publicly on a social media profile, business directory, or other web page doesn't automatically create a lawful basis to collect, store, and process that information at scale. Legitimate interests, consent, and other GDPR bases can potentially apply in specific circumstances, but these must be genuinely assessed — not assumed. Processing scraped personal data without a valid lawful basis is a GDPR violation enforceable by data protection authorities.
What did hiQ v. LinkedIn decide about web scraping?

The hiQ Labs v. LinkedIn case — decided multiple times by the 9th Circuit Court of Appeals between 2019 and 2022 — ruled largely that scraping publicly accessible LinkedIn profile data (data that doesn't require authentication to access) does not constitute unauthorized access under the Computer Fraud and Abuse Act. The decisions supported the position that publicly available data can generally be scraped without criminal CFAA liability. However, the case involved specific facts, a specific jurisdiction, and a specific statute — it doesn't create universal permission to scrape any public web page in all contexts, and other legal frameworks (ToS, data protection law, database rights) apply independently.
Does web scraping violate copyright?

It depends on what's scraped. Copyright protects creative expression — articles, photographs, artistic works, original writing — but not facts, data, or information in itself. Scraping factual data (prices, addresses, dates, numerical data) generally doesn't infringe copyright. Reproducing substantial portions of copyrighted articles, capturing and redistributing protected images, or copying other creative works may infringe copyright even when done through automated means. The distinction between scraping data and reproducing content is the key copyright analysis.