PatronView's owner details a year of fighting scrapers: 214:1 bot-to-human page loads, 35K Claude crawls per referred visitor, and Amazon's bot referred none
A year of fighting scrapers on my 1.5 million-page website. What I tried, how I failed, and what's working now. August 7, 2026
Context & Ripple Effects
PatronView's figures put a concrete workload behind a conflict publishers had already been trying to manage through robots.txt restrictions on AI bots and newer anti-scraping tools. The reported failures matter because a 1.5 million-page site offers crawlers a large surface area to revisit.
The traffic imbalance also sharpens the case for mechanisms such as Cloudflare's Pay per Crawl launch, which paired crawler charging with default blocking for new sites. PatronView reports that Amazon's bot sent no referrals, while Claude generated 35,000 crawls for each referred visitor.
First-order effects
- PatronView must absorb or block bot activity that it says outnumbers human page loads 214 to 1, making scraper mitigation an operational priority for its owner.
- Claude's reported crawl-to-referral ratio and Amazon's lack of referrals leave PatronView with little visible audience return for the bot traffic it identifies.
Second-order effects
- AI crawler operators face stronger incentives to provide publishers with clearer controls and measurable exchange for access as sites compare crawl volume with referred traffic.
- Anti-scraping and access-control providers gain a more immediate use case from publishers whose robots.txt rules or earlier defensive attempts have not adequately constrained crawlers.
Third-order effects
- If publisher reports repeatedly show heavy crawling without referral value, access to web content is likely to move toward enforceable publisher controls and paid or negotiated crawler access rather than voluntary bot rules alone.
The trend: Publisher-AI search relations are shifting from permissive crawling toward verifiable access controls tied to the value crawlers return.