"New data is reinforcing a structural shift in how AI systems access publisher content: AI models are increasingly scraping publisher content, regardless of bot-blocking measures or content licensing deals meant to control usage, improve attribution or drive referral traffic. New research from analytics firms and bot-tracking companies shows AI tools are increasingly crawling publisher sites as inputs for AI-generated summaries and training, while sending back only limited referral traffic."
""Despite the investment in cybersecurity, publishers are structurally on the back foot. Most sites can only use one cybersecurity provider due to cost and latency. However, AI companies have dozens of scraping tools they can use interchangeably that all evolve quickly to get around those defenses," said Toshit Panigrahi, co-founder and CEO of TollBit. However, Panigrahi also described these trends as an opportunity, as they show the increasing demand for web content."
"The composition of web traffic is changing as a result of all this AI bot scraping, meaning human visits to websites are decreasing as AI bot traffic increases. According to TollBit's latest "State of the Bots" report, AI bot scraping in the second half of 2025 grew 29 percent from Q2 to Q3, and 20 percent from Q3 to Q4 in 2025."
AI models increasingly ingest publisher content by crawling sites for summaries and training, often bypassing bot-blocking measures and licensing controls. This scraping behavior returns limited referral traffic, turning publisher content into infrastructure for AI products rather than reader destinations. The balance of web traffic is shifting toward bots, reducing human visits and referral-driven revenue. Publishers face defensive limits because most can afford only a single cybersecurity provider, while AI companies deploy many rapidly evolving scraping tools. The increased scraping indicates high demand for web content, creating potential monetization opportunities if publishers can capture value from that demand.
Read at Digiday
Unable to calculate read time
Collection
[
|
...
]