Artificial intelligence
fromTheregister
1 day agoPublishers say no to AI scrapers, block bots at server level
Millions of websites are blocking AI crawler bots via robots.txt to prevent training-data scraping and reduce non-human server traffic.
More than 70 companies gathered for the workshop, roughly half of which were publishers - a handful from Europe. The rest were a mix of big tech representing their respective LLMs, tech vendors and cloud edge companies Cloudflare and Fastly, who are now taking a far more active role in helping publishers block unauthorized bots, shifting from background tech enablers to vocal gatekeepers in the AI era.