A volunteer project by a group of libraries, End of Term Presidential Harvest 2016, is saving valuable pages on federal government sites at risk of vanishing
Jim Dwyer / New York Times : Tweets: @wcronon , @sjsuischool , and @edsu Tweets: Bill Cronon / @wcronon : The Web is the most important historical document of our age. Here's why historians worry about its future: http://www.nytimes.com/... SJSU iSchool / @sjsuischool : “With the arrival of any new president, vast troves of information on government websites are at risk of vanishing within days.” http://twitter.com/... Ed Summers / @edsu : Some well deserved credit to @vphill and others for their efforts over the years to archive end of term websites http://www.nytimes.com/...
Context & Ripple Effects
The End of Term Presidential Harvest 2016 is a volunteer effort by libraries racing an administrative clock: with each change of president, vast troves on federal websites can vanish within days, so the project crawls and snapshots those pages before they disappear. Historians like Bill Cronon frame the stakes plainly — the web is the primary historical document of this era, yet no one officially owns preserving it.
The harvest sits early in a now-visible arc of independent preservation: the Internet Archive grew from 2TB in 1997 to roughly 100PB by 2022 (its scale-up), explored blockchain-based tamper-proof storage after worries about meddling, and by 2025 had cataloged some 73,000 US government webpages the Trump administration had expunged — evidence the risk the 2016 volunteers mobilized against was not hypothetical.
First-order effects
- Federal agencies changing their web presence at transition time lose the default guarantee that old pages persist; the volunteer harvest captures them instead, handing historians and researchers a pre-transition record of climate, data, and policy pages.
- The libraries involved — with SJSU iSchool amplifying the effort — establish a repeatable playbook of coordinated crawling that did not depend on any single institution's budget or mandate.
Second-order effects
- Independent archives become the load-bearing layer: the Internet Archive's growth to ~100PB positions it as the de facto custodian when governments themselves remove or alter content.
- Because a sitting administration can delete pages after they are archived, preservation projects face pressure to make copies tamper-resistant — the concern behind the Internet Archive's exploration of blockchain-backed, meddling-proof storage.
Third-order effects
- The pattern has already repeated at larger scale: by 2025 the Internet Archive had cataloged ~73,000 expunged government webpages, suggesting end-of-term harvesting hardens into routine civic infrastructure activated by every transition rather than a one-off rescue.
- Preservation logic spreads beyond government sites — projects like Perma attack the same fragility in scholarly citations via link rot, pushing toward a norm that enduring documents carry permanent, independently archived references rather than trusting publishers or platforms.
The trend: Web archiving is consolidating into standing independent infrastructure that activates around each presidential transition, because neither governments nor platforms can be relied upon to preserve the public record themselves.