OpenAI details GPTBot, which the company uses to crawl the web for data to improve its AI models, and how site admins can opt out using robots.txt
GPTBot is OpenAI's web crawler and can be identified by the following user agent and string. Emilia David / The Verge : Now you can block OpenAI's web crawler Barry Schwartz / Search Engine Land : GPTBot: OpenAI releases new web crawler Barry Schwartz / Search Engine Roundtable : OpenAI's ChatGPT New Web Crawler - GPTBot Kai Xiang Teo / Insider : Adding one line of code can now prevent OpenAI from accessing a website's data to train ChatGPT Onur Demirkol / gHacks Technology News : OpenAI reveals new web crawler named “GPTBot” Lacton Muriuki / Cryptopolitan : OpenAI's GPTBot takes on the web - what you need to know Jose Antonio Lanz / Decrypt : OpenAI to Unleash New Web Crawler to Devour More of the Open Web Charles W. Bailey, Jr / Digital Scholarship : OpenAI's New Web Crawler: GPTBot Laurie Sullivan / MediaPost : OpenAI Releases GPTBot Web Crawler That Marketers Can Block Luke Jones / WinBuzzer : OpenAI Launches GPTBot Webcrawler to Train ChatGPT, but Website Owners Can Block It Sarvesh Mathi / MediaNama : Here's how you can block OpenAI's web crawler from scraping your site Brayden Lindrea / Cointelegraph : OpenAI launches web crawler ‘GPTBot’ amid plans for next model: GPT-5 Ben Thompson / Stratechery : Apple Earnings; OpenAI, GPTBot, and Robots.txt; Zoom's Terms-of-Service Meera Navlakha / Mashable : OpenAI launches webcrawler GPTBot, and instructions on how to block it Richard Speed / ITPro : OpenAI quietly unveils GPTBot dedicated web crawler Threads: Sam Lessin / @lessin : This is the key - unless OpenAI can figure out a justification for allowing their crawler, anyone with any IP or information would be crazy to not just fully block them. Dare Obasanjo / @carnage4life : It is now straightforward for websites to opt out of being used by OpenAI to train ChatGPT because they've published the identifying information of their web crawler. I'm curious to see if many... Mastodon: Karl Dubost / @karlcow@mastodon.cloud : So here the interesting question that robots.txt doesn't solve with regards to chatgpt. Maybe we also need a meta for html content that can be attached to content. — The scenario is you might forbid a specific bot to parse your content through robots.txt but it doesn't prevent a secondary parsing through a third party … X: Robert Hansen / @rsnake : If you have a desire to block ChatGPT you can do it with the following signatures. Keep in mind that won't block other bots that scrape you that they can scrape. AI will get what it “wants”: https://platform.openai.com/ ... Shubham Saboo / @saboo_shubham_ : Breaking 🚨 OpenAI just launched GPTBot, a web crawler designed to automatically scrape data from the entire internet. This data will be used to train future AI models like GPT-4 and GPT-5! GPTBot ensures that sources violating privacy and those behind paywalls are excluded. [image] Max Willens / @maxwillens : Going to be really interesting to see how many web publishers, particularly those in the news space, decide to block this guy from crawling their sites https://www.seroundtable.com/ ... Duane Forrester / @duaneforrester : Not a perfect solution, but it's a start. At least it helps us control one robot. Open AI has released their user agent and string allowing us to “control” GPTBot in robots.txt if desired: https://platform.openai.com/ ... @clarkesworld : We are now blocking another one of OpenAI's scraping bots. You can too. (I don't know if this is the secret one we couldn't block before or if that one is still in use.) https://searchengineland.com/ ... Justin Jackson / @mijustin : 👀 now you can add this to your robots.txt file: User-agent: GPTBot Disallow: / https://platform.openai.com/ ... Brodie Clark / @brodieseo : It's about time... You're now able to use robots.txt to prevent the crawling of your website content by OpenAI. There's plenty of publishers out there that will be glad to see this introduced, considering how bad attribution is in ChatGPT. h/t @badams https://platform.openai.com/ ... [image] Gergely Orosz / @gergelyorosz : It's now possible to block ChatGPT's crawler on any website you control. Added the block: it's a no-brainer. Why? ChatGPT cites no sources. It's a one-way relationship where OpenAI takes what is published on the internet, and then doesn't give a single reference or link back. [image] Gergely Orosz / @gergelyorosz : Details on how to block content: straight from OpenAI: https://platform.openai.com/ ... So I assume they will adhere to this: but *only* going forward. They scraped the public internet already, without notifying anyone (they didn't need to), and won't (and cannot!) remove those parts. [image] @therealrutledge : Looks like publishers now have a way to block OpenAI (ChatGPT) from crawling our sites and using our data. Will be enabling later today. https://platform.openai.com/ ... Thomas Baekdal / @baekdal : I just blocked it from my site.
Context & Ripple Effects
This is an early formalization of a publisher control point for generative AI: OpenAI identifies a dedicated crawler and routes exclusion through the web’s established robots.txt convention. Within weeks, major news publishers were blocking GPTBot from their content, showing that the control was immediately consequential.
The later coverage shows the limits of a naming-and-directives approach. Google introduced a separate training opt-out while preserving search visibility, and Cloudflare later offered network-level blocking for AI-training bots; meanwhile, some publishers encountered newly named Anthropic bots after blocking earlier ones.
First-order effects
- Site administrators can distinguish OpenAI’s training crawler from ordinary traffic and signal that it should not access their sites through robots.txt.
- OpenAI gains a publicly identifiable collection channel, while publishers must actively configure and maintain their own exclusion preferences.
Second-order effects
- Publishers can apply different rules to search indexing and model training, a distinction reinforced by Google’s later separate AI-training opt-out. That makes crawler policy a routine part of publishing operations rather than a binary choice to be visible online or not.
- Bot operators face stronger incentives to document user agents and crawling behavior as sites and infrastructure providers turn those identifiers into access controls.
Third-order effects
- The web’s default openness is being renegotiated into granular, publisher-managed permissions for AI ingestion; robots.txt remains a lightweight signal rather than a technical guarantee.
- If crawler traffic continues to impose operational costs—as illustrated when OpenAI crawler activity overwhelmed an e-commerce site—enforcement is likely to shift further from voluntary directives toward hosting and network-layer controls.
The trend: AI training is turning web crawling from an invisible background function into a contested, configurable layer of digital publishing and infrastructure.