Google has launched GoogleOther, a new web crawler.
GoogleOther will be used internally by various Google teams to crawl the public web, said Google’s Gary Illyes on LinkedIn.
It respects the same directives and protocols of the main Googlebot crawler and will free up some resources for the main Googlebot crawlers.
GoogleOther. Google added GoogleOther to its crawlers documentation:
“Generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. The GoogleOther crawler always obeys robots.txt rules for its user agent token and the global user agent (*), and uses the same IP ranges as Googlebot.”
User Agent. The user agent token and full user agent string are simple, it is just “GoogleOther”.
Why the new bot? This new bot will “ultimately will take some strain off of Googlebot,” Illyes explained:
- “As we optimize how and what Googlebot crawls, one thing we wanted to ensure is that Googlebot’s crawl jobs are only used internally for building the index that’s used by Search. For this we added a new crawler, GoogleOther, that will replace some of Googlebot’s other jobs like R&D crawls to free up some crawl capacity for Googlebot.”
Google uses it for internal purposes. Does that mean it is being used for Bard? It is unclear.
Should you block GoogleOther? It is unclear what would happen if you try to block this new bot. But I suspect you should keep an eye on it and if it is causing too much stress on your server, you can decide if you want to allow or deny access to your site for this new Googlebot.
Why we care. Many of you check your crawling activities and bot activity on your website and in your log files. When you see this new GoogleOther crawler, do not be alarmed. It is a real Googlebot.
Source link
