Internet infrastructure company Cloudflare has implemented a new default setting that allows publishers and content creators to block unauthorised AI scraping and use of original content.

From Wednesday, a new permission-based model will require AI companies to obtain explicit authorisation from a website before scraping, allowing website owners to choose whether they want AI crawlers to access their content and to decide how AI companies can use it.

The company is the first internet infrastructure provider to block access to content by AI crawlers by default without permission or compensation.

When signing up with Cloudflare, each new domain will be asked whether it wants to allow access to AI crawlers, giving customers the option to choose in advance whether to explicitly allow or deny access to AI crawlers.

Customers will be able to check their settings and enable crawling at any time if they want their content to be freely accessible.

Currently, AI crawlers collect content such as text, articles and images to generate responses, without sending visitors back to the original source, depriving content creators of revenue and the satisfaction of knowing that someone is viewing their content, Cloudflare said.

In a statement, Matthew Prince, co-founder and chief executive of Cloudflare, emphasised the need to give publishers the control to provide a new economic model that works for creators, consumers, AI founders and the future of the web.

“Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it,” he added. “AI crawlers have been scraping content without limits.”

Many publishers have already praised the move.Roger Lynch, chief executive of Condé Nast, described the new tool as a “gamechanger” for publishers, setting a new standard for online content compliance.

“When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,” he said.

Neil Vogel, chief executive of Dotdash Meredith, the largest publisher in the US comprising over 200 publications including USA today, also praised the initiative.

“As our industry faces these challenges, we are optimistic the Cloudflare technology will help combat the theft of valuable IP,” he said.

Bill Ready, chief executive of Pinterest, said the new setting will help build “a healthy internet infrastructure” where content is used for its intended purpose, enabling creators and publishers to grow.

---