The UK's Independent Publishers Alliance urges members to block OpenAI and Google crawling, as OpenAI extends ChatGPT's training database beyond September 2021
Context & Ripple Effects
The alliance’s call came as publishers were beginning to treat crawler access as a distinct control point: several major news outlets had already blocked OpenAI’s GPTBot, establishing a practical route for withholding content from model collection.
The dispute later broadened from voluntary blocking to market conduct. The alliance’s subsequent antitrust complaint over Google’s use of web content for AI Overviews shows why the early crawl-control debate mattered to independent publishers.
First-order effects
- Alliance members are encouraged to deny OpenAI and Google crawler access, potentially reducing the material those systems can collect from participating publishers.
- Publishers must make an immediate trade-off between restricting AI reuse and preserving whatever discovery or referral benefits may accompany crawler access.
Second-order effects
- The move increases pressure on OpenAI and Google to distinguish training and AI-use crawling from services publishers may still want, rather than treating access as all-or-nothing.
- A coordinated publisher stance gives smaller outlets a shared negotiating position on the terms under which their content can be collected and used.
Third-order effects
- If crawler blocks become widespread, web-content access could shift from an informal default toward explicit permissions, licensing, and product-level opt-outs.
- The central competition question becomes whether publishers can refuse AI reuse without sacrificing visibility in dominant discovery channels—a concern later reflected in the UK CMA’s proposed publisher opt-outs for Google AI products.
The trend: This is an early instance of publishers seeking granular control over how AI platforms collect, train on, and surface their content.