In short: web scraping still works and is still legal for public, logged-off data, but the ground moved this year. Cloudflare blocks Agent and Training crawlers by default on new domains from September 15, 2026. Courts have shifted from arguing about the data to arguing about the method, with DMCA circumvention claims surviving dismissal in July. And the Ninth Circuit held in August that when an agent fetches a page, the user directing it is the one accessing the site. The practical upshot: scrape when the data is genuinely site-specific, and use an extraction endpoint when it is not.
Web scraping is the automated extraction of data from websites: a program requests a page, parses the HTML, and pulls out the parts you want. It has been the default answer to "I need data from a site that has no API" for twenty years.
That default is breaking down, and 2026 is the year it became obvious. The web is closing to automated access, the legal theory shifted from what you collect to how you collected it, and AI agents changed who is on the hook.
This guide covers how scraping actually works, what specifically changed this year, where the law now sits, and the question worth asking before you build a scraper at all.
A note on scope: this is not a guide to bypassing bot protection. Given where the case law is heading, that section would be a liability rather than a service. What follows is what the defenses are, why they exist, and what your options are.
01How Web Scraping Works
Four steps, and every scraper is some version of them.
Fetch. Request the page over HTTP. Simple with requests or fetch, harder when the site requires a real browser.
Render. Many sites build their content in JavaScript after the initial HTML loads. If so, you need a headless browser like Playwright or Puppeteer rather than a plain HTTP client. This is where scraping stops being cheap.
Parse. Find the data inside the markup, using CSS selectors, XPath, or a library like BeautifulSoup. This is the part you maintain forever, because it breaks whenever the site changes its layout.
Store. Write it somewhere useful, deduplicated, in a schema you defined.
The fetch and store steps are stable. Render and parse are where the cost lives, and both got significantly more expensive this year.
02What Changed in 2026
The web is closing to automated access
Cloudflare sits in front of a large share of the web and has moved from a simple allow-or-block model to classifying bots by function: Search, Agent, and Training. Site owners can now apply different policies to each.
The date that matters: from September 15, 2026, new domains onboarding to Cloudflare get defaults that block Training and Agent crawlers on pages displaying ads, while Search crawlers remain allowed. Mixed-purpose crawlers that combine Search and Training are blocked by all AI-training configurations.
That is a meaningful shift. It sorts bots by what they do with the content afterwards rather than what they claim to be, and the default for new sites is now "no" for a large category of automated access.
Alongside it, Cloudflare introduced Pay-Per-Crawl, which returns HTTP 402 and asks crawlers to pay for access rather than blocking them outright. Large numbers of sites now disallow AI training in robots.txt, though robots.txt remains a request rather than an enforcement mechanism.
The legal theory moved from the data to the method
This is the change most teams have not registered, and it is the one that matters.
For years the argument was about the data: is this copyrighted, is this fair use, is this public. The reference point was hiQ Labs v. LinkedIn, where the Ninth Circuit affirmed an injunction letting a data analytics company keep scraping public LinkedIn profiles, finding hiQ had raised serious questions about whether the Computer Fraud and Abuse Act reaches information that requires no authorization to view. The Supreme Court vacated and remanded in light of Van Buren, and the Ninth Circuit affirmed again in April 2022.
The detail usually left out of the summary matters: hiQ still lost. In November 2022 the district court found hiQ had breached LinkedIn's User Agreement, and the parties settled. The CFAA claim failed and the contract claim did not. That pattern, statutory claims failing while terms-of-service claims survive, has held up since.
The 2026 cases argue something different again, under the DMCA's anti-circumvention provision, Section 1201.
On July 31, 2026, Judge Paul Engelmayer largely denied motions to dismiss in Reddit's suit against Perplexity and SerpApi, in a 63-page opinion allowing the Section 1201(a) circumvention claims to proceed against both. The court treated Google's SearchGuard anti-scraping technology as an access control, and found Reddit had plausibly pleaded that Perplexity conspired with data scrapers to bypass access controls.
The significance is that Section 1201 targets the act of bypassing a technical protection measure, independent of what you do with the data afterwards. Fair use does not answer it, because fair use is about the use and this is about the access. If anti-bot systems are technological protection measures, then defeating one is the violation regardless of whether the data was public.
Nothing is decided. Surviving a motion to dismiss is not winning. But the theory now has a judge willing to let it reach discovery, which is further than it had got before, and it makes "we rotate residential proxies and spoof TLS fingerprints" a worse thing to have written down than it was two years ago.
Agents changed who is doing the accessing
On August 4, 2026, the Ninth Circuit decided Amazon.com Services, LLC v. Perplexity AI, Inc., No. 26-1444, vacating a preliminary injunction that had barred Perplexity's Comet browser assistant from Amazon.
The holding: when a user tasks an agent with acting on their behalf, it is the user who accessed Amazon's computers, not Perplexity. Amazon was therefore unlikely to succeed under either the CFAA or California's CDAFA. The panel stressed that the CFAA is principally an anti-hacking criminal statute and that courts should be cautious about extending it to new technologies.
The facts are worth knowing, because they were not favourable. Amazon showed that Comet had navigated password-protected pages, reached private account data, and sent it to Perplexity's servers, that Comet used no distinguishing user-agent string, and that Amazon had sent a cease-and-desist revoking authorization. The district court had granted the injunction on those facts in March 2026. The Ninth Circuit still vacated it.
This is not a licence for anything. The case was remanded, contract and terms-of-service claims are untouched, and one circuit is not settled law. But it is the first appellate answer to who is accessing a site when an agent does the fetching, and it points at the person giving the instruction. Good summaries from Cooley and Wilson Sonsini if you are building anything agentic.
The volume changed
AI crawler traffic has grown to a scale that made the old permissive posture untenable for site operators, and a meaningful share of it disguises itself as ordinary browser traffic rather than declaring what it is.
That is why the defaults changed. Site operators are not reacting to your scraper specifically. They are reacting to aggregate load, and your request arrives inside it.
03Is Web Scraping Legal?
Not legal advice, and jurisdiction matters enormously. What can be said accurately:
Scraping public, logged-off data has fared well in US courts on statutory claims. hiQ v. LinkedIn is the reference point.
Scraping behind a login is a different question. Terms of service claims survive even where CFAA claims fail, and account bans for logged-in scraping are routine.
Circumventing technical protection measures is the live risk in 2026. Reddit's Section 1201 claims against Perplexity and SerpApi survived dismissal on July 31, 2026 and are heading to discovery. That theory sidesteps fair use entirely.
Personal data brings GDPR into it, independent of any of the above. Scraping names, emails, or profiles of EU residents needs a lawful basis, and "it was public" is not one on its own.
The practical guidance most sources converge on: collect only public logged-off data, do not circumvent access barriers, rate limit respectfully, check robots.txt, read the terms, and document what you collected and from where.
04Web Scrapers, Data Scrapers, and Scraper APIs
The tooling splits into three rough categories, and the names get used interchangeably in ways that confuse the choice.
A web scraper you write yourself. Python with requests and BeautifulSoup for static pages, Scrapy for anything with scale, Playwright or Puppeteer when JavaScript rendering is required. Maximum control, zero cost beyond your own time, and you own every layout change and every block.
A data scraper as a product. Browser extensions and desktop tools that point at a page and pull a table out of it. Genuinely useful for one-off work and for people who do not write code. They inherit all the same blocking problems and generally cannot be scheduled or scaled.
A scraper API. A hosted service that handles the fetch, rotates proxies, solves challenges, and returns HTML or parsed data. You are paying someone else to run the arms race. That is a legitimate trade, though note that the 2026 circumvention cases are aimed squarely at the part of the stack these services sell.
The fourth option is the one this post is about, and it is different in kind rather than degree. Instead of a scraper API that fetches a page for you, an extraction endpoint returns the specific thing you wanted: the readable text, the sitemap, the metadata, the screenshot. No page to parse, no selectors to maintain, no challenge to solve, because the endpoint operator is not fighting the site.
05The Question Worth Asking First
Given all of the above, the useful question is not how to scrape. It is whether the thing you want requires scraping at all.
Scraping exists because a site has data and no interface for getting it. But a large share of what teams scrape is not site-specific data. It is a general capability that happens to be visible on a page.
Some examples of the difference:
| What you actually want | Scraping approach | Alternative |
|---|---|---|
| The readable text of an article | Fetch, render, guess at the content selectors, strip nav and ads | A content extraction endpoint |
| Every URL on a site | Crawl the site, follow links, deduplicate | Parse the sitemap |
| Structured product or page data | Parse markup, maintain selectors per site | Read the embedded schema.org data |
| Which links on a page are dead | Crawl, request each link, track responses | A link checker endpoint |
| What technology a site runs | Fingerprint headers and scripts yourself | A tech detection endpoint |
| A screenshot of a page | Headless browser, manage the browser pool | A screenshot endpoint |
| Company or domain information | Scrape an about page and hope | WHOIS and domain data |
In every row on the right, someone else owns the maintenance. That is the actual trade, and it is the same trade as any other build-versus-call decision: you keep the part that differentiates you and stop owning the part that does not.
06When You Genuinely Do Need to Scrape
Being honest about this matters, because the answer is not always an API.
The data is specific to one site and nobody else has it. Competitor pricing on a particular retailer. Listings on a niche marketplace. There is no generic endpoint for that, because the value is in that site specifically.
The volume is high and the target is stable. If you are pulling the same well-structured site daily and it rarely changes layout, a scraper is cheap to run and cheap to keep working.
You need everything, not a field. Archival, research corpora, and dataset construction want the whole page, not an extraction of it.
No alternative exists. Sometimes there just is not an endpoint. That is a legitimate reason.
In those cases, scrape carefully: respect robots.txt, rate limit, identify your bot honestly in the user agent, cache aggressively so you fetch each page once, and stay on public logged-off content.
07When You Do Not
The case against scraping is strongest when what you want is a general capability rather than one site's data.
Article and content extraction. Every site structures its content differently and every one changes it. A parser tuned to one site's markup breaks on the next. Extract Text from Webpage and AI-ready Clean Data Extractor return readable content with navigation, scripts and markup already removed.
Site structure. Crawling a site to map it is expensive and looks exactly like the traffic bot defenses are built to stop. Extract Sitemap from URL reads what the site publishes about itself, which is faster, politer, and more accurate.
Link and health checking. A broken link checker and a link extraction endpoint do this without you managing a crawl queue.
Domain and company data. WHOIS lookups, domain age, and tech stack detection come from registries and fingerprinting rather than from parsing an about page.
Screenshots. Running a headless browser pool to capture pages is real infrastructure. A screenshot endpoint is a call.
All of these sit in the ApyHub catalog on one key, with the free plan allowing 5 calls a day and no credit card, which is enough to compare output against what your current scraper returns.
Browse data extraction endpoints
08Scraping and AI Agents
Two things are worth separating here.
Agents that scrape. An agent fetching raw HTML burns an enormous amount of context on markup that carries no information. A page that is 4,000 tokens of HTML might be 400 tokens of actual content. Multiply that across a research task touching twenty pages and the difference is the whole context window. Structured extraction endpoints return the content without the markup, which is why they suit agent workflows better than a raw fetch does.
Agents as the thing being blocked. Cloudflare's new taxonomy explicitly separates Agent traffic from Search traffic, with different default policies. If your product involves an agent browsing on a user's behalf, you are now in a category that some sites block by default. Building on endpoints rather than fetches sidesteps that entirely, because the endpoint operator has the relationship with the source.
Every ApyHub endpoint is available over MCP, so an agent can discover and call extraction capabilities directly rather than fetching and parsing pages itself.
09A Decision Guide
| Situation | Approach |
|---|---|
| Data specific to one site, no alternative | Scrape, carefully and politely |
| Readable content from arbitrary URLs | Content extraction endpoint |
| Site structure and URL discovery | Sitemap parsing |
| Product or page metadata | Structured data extraction |
| Domain, company or tech information | Domain intelligence endpoints |
| Visual capture of a page | Screenshot endpoint |
| Anything behind a login | Get permission, or do not |
| Anything requiring bot-detection bypass | Reconsider, given the 2026 case law |
10Conclusion
Web scraping is not dead and it is not illegal. It is getting narrower.
The set of problems where scraping is the right answer is shrinking, squeezed from two sides: technically by defenses that make it expensive, and legally by a shift toward treating circumvention as the violation. What is left is genuinely site-specific data with no alternative, and that is a smaller set than most scraping projects assume.
For everything else, the question is whether you want to own a parser for a website you do not control. In 2026 that is a worse deal than it used to be.
11FAQ
What is web scraping?
Web scraping is the automated extraction of data from websites. A program requests a page, parses the HTML, and pulls out specific information. It is typically used when a site holds data you need and offers no API for accessing it.
Is web scraping legal in 2026?
Scraping public, logged-off data has fared well in US courts, notably hiQ Labs v. LinkedIn, though hiQ still lost on breach of LinkedIn's user agreement. The live risk in 2026 is different: on July 31, 2026 a federal judge allowed Reddit's DMCA Section 1201 circumvention claims against Perplexity and SerpApi to proceed, on the theory that bypassing bot detection is itself the violation regardless of what the data is used for. Personal data separately brings GDPR into scope. This is not legal advice and jurisdiction matters.
What changed for web scraping in 2026?
Three things. Cloudflare moved to classifying bots by function, and from September 15, 2026 new domains block Training and Agent crawlers by default on pages with ads. Litigation shifted from arguing about the data to arguing about the method of access: on July 31, 2026 Reddit's DMCA Section 1201 circumvention claims against Perplexity and SerpApi survived dismissal. And on August 4, 2026 the Ninth Circuit held in Amazon v. Perplexity that a user directing an agent is the party accessing the site under the CFAA, not the agent's operator.
Does Cloudflare block web scrapers?
Cloudflare Bot Management combines TLS fingerprinting, IP reputation scoring, JavaScript challenges, behavioural analysis, and interactive challenges. It classifies bots as Search, Agent, or Training and allows site owners to apply separate policies to each. From September 2026, new domains block Agent and Training traffic by default on pages displaying ads.
What is the difference between web scraping and an API?
Scraping pulls data out of a page designed for humans, which means parsing markup and maintaining that parser as the site changes. An API returns structured data through an interface designed for programs, with a documented schema and a stable contract. Scraping is what you do when no API exists.
What is the difference between web scraping and web crawling?
Crawling is discovering and visiting pages, typically by following links. Scraping is extracting data from a page once you have it. Most projects do both, though reading a site's sitemap is usually a better way to discover URLs than crawling it.
Do I need a headless browser to scrape?
Only if the content is rendered by JavaScript after the initial HTML loads. If the data is in the raw HTML response, an HTTP client is far faster and cheaper. Check the page source before reaching for Playwright or Puppeteer.
Why does my scraper keep breaking?
Almost always because the site changed its markup. Scrapers depend on selectors that target a specific page structure, and that structure is not a contract. Any redesign, A/B test, or component change can break the parse. This is the maintenance cost people underestimate when choosing scraping over an endpoint.
What is a scraper API?
A scraper API is a hosted service that performs the fetch for you: it rotates proxies, manages browser fingerprints, handles challenges, and returns the page or parsed data. You are outsourcing the blocking arms race rather than avoiding it, which is worth noting given that the 2026 circumvention litigation targets exactly that layer.
What is the best web scraper?
It depends whether you are writing one or buying one. For code, Scrapy handles scale, Playwright handles JavaScript rendering, and requests plus BeautifulSoup covers static pages. As products, browser-extension data scrapers suit one-off extraction, and scraper APIs suit scheduled collection at volume. If what you need is a general capability rather than one site's data, an extraction endpoint avoids the category entirely.
Is there an alternative to web scraping?
Often. If what you want is a general capability rather than one site's specific data, an extraction endpoint usually exists: readable content from a URL, a sitemap, a screenshot, link status, domain information, structured page metadata. Scraping remains the right answer when the value is genuinely in one particular site.
How do AI agents affect web scraping?
Two ways. Agents fetching raw HTML waste large amounts of context on markup, so structured extraction suits them better. And agent traffic is now a distinct category in bot classification systems, subject to its own default policies, so an agent browsing on a user's behalf may be blocked where a search crawler would not be.
How do I scrape a website responsibly?
Read robots.txt and respect it, rate limit rather than requesting as fast as possible, identify your bot honestly in the user agent, cache so you fetch each page once, stay on public logged-off content, and avoid circumventing technical access controls. Document what you collect and from where.
12Sources
- Block AI Bots, Cloudflare developer documentation, on the September 15, 2026 default changes
- Cloudflare changes AI crawler access rules, Help Net Security, July 2026, on the Search / Agent / Training taxonomy
- hiQ Labs v. LinkedIn, Ninth Circuit, 2019, and the 2022 opinion on remand
- Ninth Circuit Rules on AI Agent 'Access' to Third-Party Websites Under CFAA, Cooley, August 2026, on Amazon v. Perplexity
- Ninth Circuit Addresses CFAA and Agentic AI Tools, Wilson Sonsini, August 2026
- Ninth Circuit Lifts Restrictions on Agentic AI Accessing Amazon, Eric Goldman's Technology & Marketing Law Blog
13About ApyHub
ApyHub is a curated API catalog for developers, teams, and AI agents. On the web data side it covers content and data extraction, domain and site intelligence, file conversion, data validation, and AI processing, across 20 categories in total. One subscription covers the whole catalog, billed in atoms, with headroom pooled across every API rather than locked to individual services.
Every service carries machine-readable certification covering data handling, retention, and standards alignment including GDPR, SOC 2, and ISO 27001. Every endpoint is MCP-ready by default, so agents can discover and call them without a hand-written wrapper.
ApyHub is headquartered in Amsterdam, with offices in the Netherlands, Greece, and India, and runs on EU infrastructure. The catalog holds 450+ services and 1,500+ endpoints, with new APIs and providers onboarded continuously. The free tier allows 5 calls a day with no credit card, and every service page has an API Playground for testing before you integrate.
Publishing an API? Become a provider