A new system from Cloudflare gives millions of websites the power to block AI bots from scraping their content without permission and could soon let them charge for access via a new pay-per-crawl model.
AI Crawlers A Problem for Publishers and Creators
In recent years, the rapid growth of AI tools has sparked a battle over ownership, access, and compensation. At the centre of the controversy are “AI crawlers”, i.e. automated bots developed by companies like OpenAI, Google, and Anthropic to trawl the internet, copying data from websites to train large language models (LLMs) or power AI assistants.
For creators and publishers, the issue is that this content is often scraped without permission or compensation. Unlike traditional web crawlers used by search engines, which drive traffic back to the original source and support advertising revenue, AI bots typically use the content to generate summaries, answers or outputs directly, without crediting or linking to the sites they pulled from. This bypasses publishers entirely, cutting them out of the value chain.
The BBC, for example, recently accused US-based AI firm Perplexity of using its content without consent and demanded compensation. Similar rows have erupted in the US, with lawsuits from the likes of The New York Times, and in the UK, where artists have criticised the government over weak protections.
As Matthew Prince, co-founder and CEO of Cloudflare, put it: “AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.”
Who Is Cloudflare?
Cloudflare is one of the internet’s biggest behind-the-scenes players. The US-listed tech firm provides security, performance optimisation and content delivery services for around 20 per cent of all websites globally. That scale makes any system it deploys highly influential, and potentially industry-defining.
On 1 July, the company launched a sweeping new system that gives website owners direct control over AI crawlers. Crucially, this is now turned on by default for new Cloudflare users, meaning that unless permission is granted, AI bots will be blocked from accessing site content altogether.
The move significantly changes the rules of engagement between content owners and AI firms, and lays the groundwork for a new type of economic model.
How the New System Works
The technology uses Cloudflare’s bot detection infrastructure to identify which crawlers are trying to access a site and what purpose they’re being used for, such as AI training, inference, or chatbot search responses. It means that AI crawlers must now declare their identity and intent. This in turn gives website owners the power to choose to allow access, deny it entirely, or ask for payment via a new initiative called Pay per Crawl.
Pay Per Crawl
Pay per Crawl is an experimental marketplace currently in private beta. It allows publishers to set a price (typically a micropayment) for each individual bot crawl. The AI companies must then agree to pay if they want continued access to the site’s content. The entire process is managed by Cloudflare as the intermediary.
The system also includes transparency tools such as dashboards showing how often bots visit a site and what they are collecting. This allows publishers to differentiate between helpful crawler (e.g. those from Google Search) and AI bots that may be extracting content without driving any traffic back.
Big Names Already Backing the Block
Over one million sites are already using Cloudflare’s earlier one-click tool to block AI crawlers. With the new system, even more are expected to adopt it, especially as the default setting now blocks crawlers unless explicitly allowed.
For example, leading media companies including Sky News, The Associated Press, BuzzFeed, TIME, The Atlantic, Condé Nast, Gannett (USA Today), and Dotdash Meredith have signed on to use the technology. Many see it as a step towards restoring control over their intellectual property and creating fairer terms for their contributions to the web.
“This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable,” said Roger Lynch, CEO of Condé Nast.
Also, TIME’s COO, Mark Howard, described the initiative as “a meaningful step toward building a healthier AI ecosystem—one that respects the value of trusted content and supports the creators behind it.”
Crawling Costs and Content Control
The problem, publishers argue, is that AI firms are currently reaping huge rewards from models trained on content that they never paid for. For example, a recent analysis by Cloudflare suggests that OpenAI’s crawler, GPTBot, scraped websites 1,700 times for every referral it gave in return. In comparison, Google’s bot gave one referral for every 14 scrapes – still skewed, but not nearly as extreme.
This imbalance has prompted fears that the original economic model of the open internet, i.e. where traffic from search engines fuels revenue for content creators, is breaking down. For example, as AI assistants become more prevalent and answer users’ questions directly, fewer people click through to the source material. That threatens the sustainability of journalism, research, and creative industries.
Therefore, by introducing a payment mechanism and making bot access conditional, Cloudflare hopes to reshape the model. As the company wrote in its announcement: “If the incentive to create original, quality content disappears, society ends up losing, and the future of the Internet is at risk.”
Websites and AI Firms
For website owners, especially smaller publishers, creative professionals, and independent media, Cloudflare’s system could offer a much-needed line of defence. For example, many lack the technical resources to build their own bot detection or monetisation systems. With Cloudflare now providing this as a built-in service, it levels the playing field.
For AI companies, however, it creates a new layer of complexity and potentially, cost. While some like ProRata AI and Quora have expressed support for fair compensation models, others may be forced to rethink how they access training data or structure deals with publishers.
At the same time, AI firms that continue to ignore bot exclusion rules may now find themselves more easily blocked, routed into traps (like Cloudflare’s AI “Labyrinth” of junk content), or publicly named and shamed.
The move also puts pressure on Cloudflare’s competitors, such as Amazon Web Services, Google Cloud, and Akamai, to offer similar tools or risk falling behind in the arms race over content protection and AI ethics.
A Bet on a New Internet Economy
By launching Pay per Crawl (still in beta), Cloudflare is positioning itself as both a gatekeeper and broker of a new AI-era content economy. In doing so, it’s hoping to gain influence over how value flows between creators and AI companies, and opening the door to becoming a central payments infrastructure provider in this emerging market.
CEO Matthew Prince has even floated the idea of creating Cloudflare’s own stablecoin to support seamless micropayments at scale.
Challenges
That said, challenges remain. For example, the system only protects content hosted through Cloudflare. Critics like Ed Newton-Rex, founder of Fairly Trained, argue this is a “sticking plaster” rather than a full solution. Legal frameworks, they say, are still essential to address copyright and enforce compliance across the wider web.
Baroness Beeban Kidron, a prominent campaigner for creative rights, nonetheless praised the move as “decisive action,” saying: “If we want a vibrant public sphere, we need AI companies to contribute to the communities in which they operate.”
More broadly, the battle now turns to whether Cloudflare’s system can actually become the foundation for a fairer digital ecosystem, or whether AI firms and others will try to find ways around it.
What Does This Mean For Your Business?
For publishers, a permission-based model for AI web scraping could be the first meaningful opportunity to assert control over how their work is accessed and monetised in an AI-driven world. It gives media groups, content creators, and smaller businesses a chance to protect their intellectual property without needing bespoke technical solutions, and could eventually create new revenue streams where previously there were none. If widely adopted, it also signals a move away from the unspoken assumption that public web content is free for AI companies to exploit.
What makes this development particularly relevant is Cloudflare’s scale. With its technology touching around one fifth of the internet, its default blocking of AI bots resets the baseline. AI companies can no longer rely on passive access to build their models and must now navigate a fragmented, consent-based landscape. While this raises operational challenges for developers of AI tools, it may also encourage more formal, sustainable commercial arrangements between content owners and AI firms.
For UK businesses, the implications are twofold. On the one hand, firms producing original content, e.g. publishers, consultancies, and creative agencies, stand to gain from greater control and potential compensation. On the other, companies that rely on AI systems to summarise, synthesise or build upon external content may face new hurdles or costs. It highlights the need for businesses to understand not just how AI tools function, but where their data comes from and under what terms.
However, the effectiveness of Cloudflare’s model will depend on broad adoption and robust enforcement. The Pay per Crawl system is still in beta and, for now, limited in reach. There is also the risk that aggressive scraping bots will continue to operate outside legitimate channels or spoof identities to bypass detection. In that sense, legal backing remains a missing piece. As critics point out, a voluntary system only protects those within its walls.
Even so, the shift represents a turning point. Whether or not Cloudflare’s marketplace becomes the standard, it has created a framework that others may follow or adapt. For publishers, platforms and AI companies alike, the message is that the free-for-all era of unregulated AI scraping appears to be over. The next chapter will be defined by consent, compensation and a more negotiated relationship between those who create content and those who use it.
