Block AI Crawlers The plugin tells AI crawlers (such as OpenAI ChatGPT) not to crawl your site's content for AI training. this is done by updating the site'srobots.txt, to block common AI crawlers. An AI crawler reads a website'srobots.txtto check for non-indexed requests.
It stops these AI crawlers and bots:
- ChatGPT and GPTBot- Crawlers and web browsers used by OpenAI
- Google Extended- Crawler for Google Gemini (formerly Google Bard) AI training
- FacebookBot- Crawlers for Facebook Artificial Intelligence Training
- CommonCrawl- Compiling crawlers for datasets used to train AI models
- Anthropic AI / Claude- Crawlers used by Anthropic
- Omgili- Omgili Crawler for AI Training
- Bytespider- TikTok Crawler for AI Training
- PerplexityBot- Used by Perplexity in its artificial intelligence products
- Applebot- Apple uses to train its artificial intelligence products
- Cohere- Cohere Crawler for Artificial Intelligence Training
- DiffBot- Diffbot Crawler for Artificial Intelligence Training
- Imagesift- Imagesift Crawler for Images
Experimental meta-tagging
The plugin will also add "noai, noimageai" tags to your site's meta tags. These tags tell the AI bots not to include your content as part of their dataset. These are experimental and not yet standardized.
statement denying or limiting responsibility
Attention:While the plugin adds these tags, it is up to the crawler itself to comply with this tagging requirement.
