AI crawlers account for about 80% of AI bot traffic while AI fetchers make up the remaining 20%. Automated bots and fetchers can issue thousands of requests per minute against a single site, significantly increasing server load and operational costs. Meta's AI division is responsible for more than half of the crawlers, and OpenAI generates the majority of on-demand fetch activity. Large-scale web-application and bot-management telemetry shows automated crawlers increasingly dominate web traffic compared with humans. Poorly engineered AI bots can cause performance degradation, service disruption, and unsustainable operational burdens without improved visibility and verification standards.
Cloud services giant Fastly has released a report claiming AI crawlers are putting a heavy load on the open web, slurping up sites at a rate that accounts for 80 percent of all AI bot traffic, with the remaining 20 percent used by AI fetchers. Bots and fetchers can hit websites hard, demanding data from a single site in thousands of requests per minute.
The company's report is based on analysis of Fastly's Next-Gen Web Application Firewall (NGWAF) and Bot Management services, which the company says "protect over 130,000 applications and APIs and inspect more than 6.5 trillion requests per month" - giving it plenty of data to play with. The data reveals a growing problem: an increasing website load comes not from human visitors, but from automated crawlers and fetchers working on behalf of chatbot firms.
Collection
[
|
...
]