Cloudflare introduced "pay per crawl," enabling websites to set charges for AI crawlers during its private beta, enhancing control over bot access. By default, AI crawlers will be blocked from accessing web pages, promoting a new economic model aimed at benefiting content creators. The initiative will use existing web infrastructure for paid access. Cloudflare aims to collaborate with AI companies to verify crawler identities, giving owners control over bot interactions and ensuring fair compensation schemes depend on AI companies' participation.
Today, Cloudflare unveiled "pay per crawl," a way for websites to charge AI crawlers for access, currently in private beta, providing more control over AI bots.
Cloudflare announced it will block AI crawlers from accessing web pages by default to give publishers the control they deserve and create a new economic model.
Pay per crawl integrates with existing web infrastructure using HTTP status codes and established authentication mechanisms to allow paid content access.
Cloudflare aims to work with AI companies to verify the identity of AI crawlers and enable site owners to manage bot access effectively.
Collection
[
|
...
]