"The debate over whether to block AI bots has been simmering with publishers for more than a year, as AI crawlers proliferated and publishers weighed traffic, licensing leverage, and legal risk. There is still no clear consensus, but four publishing execs whose companies have previously taken a hard line on blocking bots told Digiday that shifts in how AI engines surface and route their content are forcing them to question whether blocking is the right call."
"Will Allen, vp of product at Cloudflare, said its data shows a continuous increase in AI bot traffic to websites. "We're seeing broadly - and also what we're hearing anecdotally - there's growth [in AI bot traffic], and it comes from desire in content for training as well as content for inference and search," Allen said. Anecdotally, Allen said he still sees publishers take an aggressive approach to blocking AI bots."
Publishers are reevaluating blanket AI-bot blocks as AI crawlers proliferate and traffic dynamics change. Executives balance traffic benefits, licensing leverage, and legal risk when deciding access policies. Shifts in how AI engines surface and route content are prompting questions about lost monetization from AI-driven referrals. Some publishers report increased traffic from Perplexity, Anthropic and ChatGPT. Cloudflare data shows a continuous rise in AI bot visits to websites. Many publishers now favor nuanced strategies over blanket bans. Publishers need to identify which bots access sites, measure frequency, and assess purposes such as training, inference, or search.
Read at Digiday
Unable to calculate read time
Collection
[
|
...
]