Google lets publishers use a robots.txt flag to opt out of the company using their data to train its AI models, while remaining accessible through Google Search
here's why Meera Navlakha / Mashable : Websites can choose to opt out of Google Bard and future AI models Vallari Sanzgiri / MediaNama : Here's how web publishers can opt out of Google crawlers scraping website data to train AI models Tyler Lee / Phandroid : Google is giving websites a choice if they want to be used for Bard AI training Katyanna Quach / The Register : Medium asks AI bot crawlers: Please, please don't scrape bloggers' musings Sara Guaglione / Digiday : Why publishers are questioning the effectiveness of blocking AI web crawlers Kristi Hines / Search Engine Journal : Google Offers Publishers Control Over Bard, Vertex AI Access Michael Kan / PCMag : Don't Want Google to Use Your Website for AI Training? You Can Now Opt Out John Callaham / Neowin : Google now allows websites to opt out for being used to train its Bard AI Cherlynn Low / Engadget : Google will let publishers hide their content from its insatiable AI Threads: Mike Murphy / @mcwm : i feel like this should be an opt-in process, rather than out https://www.theverge.com/... Dare Obasanjo / @carnage4life : Google now allows opt-out from your website being used to train their AI models while still showing up in search. Once OpenAI added the ability to block their crawler, Google had no option but to do so as well. X: @google : Teens from 13-17 in the U.S. can now sign up for Search Labs and try out generative AI in Search. See how we're prioritizing quality and safety, including new improvements to the experience ↓ https://blog.google/... Ian Linkletter / @linkletter : New robots.txt flag just dropped, allowing sites to tell Google not to use content to train AI models. There should be an open license which does the same. Forums: r/technology : Google adds a switch for publishers to opt out of becoming AI training data
Context & Ripple Effects
Google had already expanded its policy to cover publicly available information for AI training, while OpenAI had documented a separate GPTBot opt-out mechanism. This move makes crawler-level choice a direct part of Google's publisher relationship rather than an all-or-nothing decision about search visibility.
The distinction matters because publishers depend on search discovery but may not want the same material to become model-training input. It separates Google's roles as a traffic gateway and an AI-model developer.
First-order effects
- Publishers can signal that Google should exclude their content from training Bard and future AI models without asking to be removed from Google Search.
- Google must operationalize a separate permission path for AI-training crawls, while retaining normal indexing for sites that use the flag.
Second-order effects
- The control raises the practical value of robots.txt as a publisher policy tool and gives publishers a clearer basis to test or negotiate how their content is used by AI systems.
- Other model builders face pressure to offer comparably granular controls; OpenAI's earlier GPTBot opt-out makes the mechanism increasingly familiar across the market.
Third-order effects
- If widely adopted, crawl permissions could become a durable boundary between content distribution and AI reuse, rather than treating search access as blanket consent for both.
- The longer-term contest shifts to whether voluntary crawler signals are sufficient for publishers, or whether AI-content terms become more formalized through platform controls and policy.
The trend: AI platforms are moving toward separating publisher access to search distribution from permission to use the same content as model-training input.