Cloudflare has announced a new initiative called “Pay Per Crawl,” aimed at reshaping how AI companies access web content for training and other AI-related purposes.
Highlights
- New Monetization Option for Publishers: Cloudflare’s “Pay Per Crawl” lets website owners charge AI bots for web scraping, turning passive traffic into revenue streams.
- Publisher Control at the Forefront: Site owners can set per-request pricing, allow or block specific crawlers, and monitor scraping activity in real time.
- Tech Behind the Model: AI bots must include proof-of-payment in HTTP headers. If not, sites return a revived
402 Payment Requiredstatus. Authentication uses Cloudflare’s Web Bot Auth system. - Growing Publisher Frustration: Internal data shows AI bots like OpenAI’s GPTBot crawl thousands of times per user click, creating an imbalance between bot traffic and referral benefits for publishers.
- AI Blocking by Default: Starting July 1, 2025, all new Cloudflare customers will have AI crawler access blocked unless manually allowed. Over 1 million existing customers have already opted into AI-blocking tools.
- AI Companies Must Opt-In: Participation in Pay Per Crawl is voluntary for AI firms. Uncertainty remains over whether big AI players will engage with the paid model.
- Early Publisher Adoption: Media brands like TIME, The Atlantic, and Condé Nast are already leveraging Cloudflare’s AI-blocking tools amid industry-wide concerns over free scraping.
Currently in private beta, this system allows website owners to choose whether to block, allow, or charge AI crawlers on a per-request basis.
Addressing Growing Publisher Concerns Over AI Scraping
The launch comes amid rising tension between publishers and AI developers over large-scale web scraping. Many publishers have expressed frustration that their content is being used to train AI models without compensation or meaningful referral traffic in return.
Over the past year, Cloudflare has introduced tools like one-click AI bot blocking and detailed AI crawler traffic reports. With Pay Per Crawl, the company is transitioning from passive defense mechanisms to an active monetization model.
How Pay Per Crawl Works
At the core of this system is publisher control. Website owners can:
- Set individual prices for each AI crawler
- Monitor crawler activity
- Decide on access permissions
If an AI bot wants to crawl a site, it must provide proof of payment via special HTTP headers. Otherwise, the site will return a 402 Payment Required status code—a long-unused HTTP status that Cloudflare has revived for this purpose.
AI bots will also authenticate using Web Bot Auth, a digital signature-based system designed to secure the transaction process.
Cloudflare will act as the merchant of record, handling billing and payouts directly to publishers through its existing payment infrastructure.
A Growing Imbalance Between Crawls and Referral Traffic
Cloudflare shared internal data that highlights the scale of the issue:
- Google’s crawler visits websites 14 times for every one referral click sent back.
- OpenAI’s GPTBot crawls a site approximately 17,000 times per referral.
- Anthropic’s bots reportedly crawl 73,000 times for each user-directed click.
For many smaller publishers—who do not have AI licensing deals like The New York Times or Axel Springer—this discrepancy represents a significant loss of value.
Early Adoption and Default AI Blocking
Several prominent publishers, including TIME, Condé Nast, The Associated Press, The Atlantic, Adweek, and Fortune, have already embraced Cloudflare’s broader strategy of default AI crawler blocking.
Starting July 1, 2025, all new Cloudflare customers will have AI crawlers blocked by default, with access granted only if explicitly allowed. More than 1 million existing Cloudflare customers have already opted into these AI-blocking tools.
AI Firms Must Opt-In
A key limitation is that AI companies must voluntarily participate in the Pay Per Crawl marketplace. Given that many currently scrape web content for free, it remains uncertain how many will adopt the paid model.
Cloudflare remains optimistic, positioning Pay Per Crawl as a neutral, automated, and scalable system for microtransactions between AI developers and content owners.
A Future of Programmatic Web Access for AI Agents
Cloudflare envisions a future where AI agents operate within defined budgets, programmatically accessing and paying for web content in real time.
This could create new economic models for web publishing, where content behind paywalls, research papers, or niche blogs can all be monetized at the network level.
For now, however, Pay Per Crawl remains a limited beta, open only to existing Cloudflare customers and select AI companies.
Whether this approach will scale across the web remains to be seen, but Cloudflare’s initiative is one of the first major industry attempts to redefine the economics of web scraping in the AI era.
As AI-generated traffic continues to grow, the conversation around fair data use, publisher rights, and AI model training pipelines is likely to intensify.
