At least 26 of the top 100 most popular websites – and 242 of the top 1,000 – are now blocking GPTBot, the web crawler OpenAI introduced Aug. 7, according to an updated analysis.
Why we care. To block or not to block ChatGPT? That has been a big question for many SEOs because ChatGPT does not cite or link to its sources. Clearly, even more of the most popular websites have decided to block GPTBot, presumably because they don’t want OpenAI scraping their data to help train its models – at least not without compensation.
12 popular websites now blocking GPTBot. Among the new additions from the top 100 most popular sites in the past month, the majority of which publish news and information:
- pinterest.com
- indeed.com
- theguardian.com
- sciencedirect.com
- usatoday.com
- stackexchange.com
- alamy.com
- webmd.com
- dictionary.com
- washingtonpost.com
- npr.org
- cbsnews.com
One big reversal. Interestingly, Foursquare, which was blocking GPTBot last month, no longer is.
What about CCbot? Common Crawl’s web crawler is still blocked less – by just 130 websites. As a reminder, Common Crawl provides part of the training data used by OpenAI, Google and others.
- 109 of the top 1,000 websites block both GPTBot and CCbot.
Limitations. 67 robots.txt files out of the 1,000 websites were not identified/inspected as part of this analysis. (That’s why I wrote “at least” in the opening sentence.)
Originality.ai’s updated analysis. Websites That Have Blocked OpenAI’s GPTBot – 1000 Website Study
Dig deeper. Should you block ChatGPT’s web browser plugin from accessing your website?
Source link
