Max Willens

OpenAI details GPTBot, which the company uses to crawl the web for data to improve its AI models, and how site admins can opt out using robots.txt

GPTBot is OpenAI's web crawler and can be identified by the following user agent and string. Emilia David / The Verge : Now you can block OpenAI's web crawler Barry Schwartz / Search Engine Land : GPT...