Since January 2024, Wikipedia has experienced a staggering 50% rise in bandwidth usage, largely attributed to AI crawlers that scrape content en masse for model training. Unlike human users, who tend to access trending topics and can be efficiently served from cached data, AI bots query a broad range of content, including less popular articles. This shift has led to slowdowns during high-traffic events and poses significant challenges for the Wikimedia Foundation, which must constantly adapt to manage the unprecedented demands on its infrastructure.
Since January 2024, Wikimedia has seen a 50 percent increase in Wikipedia's bandwidth usage, primarily driven by AI crawlers scraping vast amounts of content.
Wikimedia is well equipped to handle peaks in traffic from human visitors, but the volume of AI scraper bot traffic is unprecedented and poses new challenges.
Collection
[
|
...
]