The Wayback Machine does not challenge takedown requests and its robot.txt policy not only allows crawling to stop but removes a given site's entire history
On the internet, there are certain institutions we have come to rely on daily to keep truth from becoming nebulous or elastic.
Context & Ripple Effects
This 2018 Gizmodo piece exposes two quiet design choices in the Wayback Machine: it never contests takedown requests, and honoring a site's robots.txt doesn't just stop crawling — it deletes that site's entire archived history. At the time, robots.txt was still treated as an informal goodwill contract between crawlers and publishers rather than anything enforceable.
That asymmetry matters more now than it did then. The same mechanism later became a back door: Reddit blocked the Internet Archive entirely after catching AI companies scraping its data through the Wayback Machine, and by 2026 Originality AI counted 23 major news sites plus Reddit blocking the Archive's crawler outright. A policy built to respect publishers' wishes ended up making the archive both fragile (one line in robots.txt erases history) and contested (its openness feeds AI scraping).
First-order effects
- Any publisher can erase its entire Wayback Machine history with a single robots.txt directive, and rights holders can remove material without ever facing a counterargument from the Internet Archive.
Second-order effects
- Because the archive's contents can be silently reshaped, parties who relied on it as a stable record — including courts, which have accepted it as legitimate evidence — face a record whose completeness depends on each site's ongoing consent.
Third-order effects
- If sites keep opting out, the web's de facto historical record fragments into whatever individual publishers allow to survive, pushing preservation debates toward whether robots.txt-style self-governance is adequate for archives at all.
The trend: Web archiving is caught between publisher control and AI-era scraping pressure, as the informal robots.txt regime strains under both deletion requests and third-party data extraction.