#web-crawling

#web-crawling

What happens when the AI companies (inevitably) encounter spam and attempts at SEO/GEO manipulation in the markdown files targeted to bots? What happens when the .md files no longer provide an equivalent experience to what users are seeing? What happens if they continue crawling those pages but actually toss them out before using the content to form a response? ...And we keep conflating "bot crawling activity" with "the bots are using/liking my markdown content?" How will we know if they're actually using the .md files or not?

Cloudflare claims that Perplexity conceals its crawling identity to circumvent website restrictions, resulting in concerns over unauthorized content scraping from various sites.

The result is a web crawler that can open headless browsers, click to expand content, traverse and extract text from a target DOM element, retry failed requests, and extract internal links for recursive crawling.

The order in the disavow file doesn't matter. We don't process the file per-se (it's not an immediate filter of "the index"), we take it into account when we recrawl other sites naturally.