Cloudflare announced that, effective July 1, 2025, it is the first internet infrastructure provider to block artificial intelligence crawlers that access content without permission or compensation by default. The move flips the default for new domains toward control: during sign-up, website owners will choose whether to allow artificial intelligence crawlers and how their content may be used. Cloudflare also said crawler operators can now declare the purpose of their bots, such as training, inference, or search, to help publishers make more informed allow or deny decisions.
The company argues the traditional web value exchange has broken down as artificial intelligence systems generate answers without referring users back to source sites, depriving creators of traffic and revenue. Cloudflare’s CEO Matthew Prince said the goal is to return control to publishers while enabling responsible innovation, framing the shift as part of building a new economic model for the web. Executives from major publishers and platforms, including Condé Nast, Dotdash Meredith, Gannett Media, Pinterest, Reddit, and Ziff Davis, voiced support for stronger guardrails and compensation frameworks for original content.
Cloudflare, which says it helps manage and protect traffic for 20 percent of the web and handles trillions of requests daily, has been escalating its controls. In September 2024, it added a single-click option to block artificial intelligence crawlers, a feature chosen by more than one million customers. Now, the company is enforcing a permission-based model that requires explicit approval before scraping, with a default “off” stance for new domains and simple settings to enable crawling later. Cloudflare is also advancing bot transparency, proposing new mechanisms for artificial intelligence bots to authenticate themselves and participating in a protocol to standardize bot identification.
Industry groups and publishers broadly welcomed the change, framing it as a pathway to fair value exchange and sustainable innovation. The release cites support from organizations across media and technology, including the Associated Press, the Atlantic, TIME, Universal Music Group, Stack Overflow, and Webflow. Cloudflare is also pointing to complementary efforts and tooling, such as managed robots controls, cryptographic bot verification within its Verified Bots Program, and a Pay Per Crawl beta that would allow content owners to charge artificial intelligence crawlers for access. Together, these moves aim to establish a more transparent, permission-based ecosystem where creators can protect, monetize, and selectively share their work.
