Edge cloud traffic analysis between April 16 and July 15 across Next Gen WAF and Bot Management shows AI-driven crawlers produce close to 80% of AI bot request volume while fetcher bots account for about 20%. Meta-originated bots contributed more than half of observed AI bot traffic, surpassing Google and OpenAI combined. Crawler bots systematically scan websites to collect content for searchable indexes or training language models, serving as a precondition to model training. Fetcher bots retrieve pages in real time in response to user actions to surface up-to-date information. Real-time fetching presents greater operational strain, with a fetcher observed at 39,000 requests per minute versus crawler peaks near 1,000.
access website content in response to user actions. For example, when a user requests up-to-date information on a specific topic, a fetcher bot retrieves the relevant page in real time. They are also used to help surface website links that match a user's search query, directing them to the most relevant content. Crawler bots contribute nearly 80% of the total AI bot request volume, with fetcher bots making up the remaining 20%.
Real-time fetching by AI bots, the report states, is a greater challenge than peak crawler rate. Analysis revealed that in one case, a single crawler reached a peak of around 1,000 requests per minute. Real-time fetching, on the other hand, is significantly more aggressive: in one instance, a fetcher bot made 39,000 requests per minute to a single website at peak load.
Collection
[
|
...
]