A significant uptick in server response times for crawling has been observed, despite a decrease in total crawl requests from Google. This issue appears to have originated earlier this month and affects various CDNs, particularly those that haven’t updated their IP ranges for Googlebot appropriately. Notably, Cloudflare has managed the change without issues. Gianna Brachetti-Truskawa highlighted the problem, while John Mueller from Google mentioned that changes to Googlebot’s IP list are published in a JSON file, which can be monitored for updates to alleviate concerns about crawling.
Have you seen a recent drop in new users, and/or found that Google's crawl rate has dropped on your site while server response times seem to be higher than usual?
Luckily, Cloudflare seems to be on top of it! But we found reports of a few websites delivered via other CDNs, including larger ones like Akamai Technologies, who run into the issue, suggesting that their CDN providers might not have updated their IP ranges for Googlebot yet.
We push the IP json files automatically -- changes happen from time to time. If you need to alert internally on those files, feel free to poll them.
Collection
[
|
...
]