Google’s 18-to-1 Search Tollbooth

A decade ago, Google crawled two publisher pages for every visitor it sent back. Cloudflare CEO Matthew Prince says the ratio is now 18 to one: Google Search needs more of the web just as its newest interface makes visiting the web less necessary.

The click was search’s settlement mechanism

The old search bargain was not written down, but its machinery made the terms visible. Crawlers copied and indexed pages, rankings organized them, and the click transferred a user to the publisher that had supplied the information. Google monetized discovery, while the publisher gained an opportunity to sell a subscription, display an advertisement, complete a purchase, or establish a direct relationship.

AI initially reinforced that design. In 2020, Google described AI in search as a way to index individual passages from webpages, allowing the engine to retrieve a more precise piece of a page. The model improved the routing layer. The destination remained somewhere else.

AI Overviews changed the completion event. Google said it would make the feature available to more than one billion users by the end of 2024, moving generated answers from a laboratory feature into a mass-market interface. Once the results page can synthesize several sources into a sufficient response, users can receive the web’s information without entering the web’s businesses.

AI Overviews appeared in 18% of a study covering roughly 69,000 searches by 900 US Google users. Conventional results remained the majority, and users still clicked links from some generated answers. But when an Overview appeared, users clicked a link in 8% of searches, compared with 15% when no Overview appeared. The feature did not have to replace every blue link to alter the exchange; it only had to make the click less necessary at the margin.

By May 2025, 69% of news searches ended without a click-through, up from 56% a year earlier. Cloudflare says human traffic to many businesses’ websites fell nearly 40% between June 2025 and April 2026 as AI search expanded.

Prince’s crawl-to-visit ratio prices the old bargain. Google can ingest more pages for every person it returns because the answer layer consumes the information before the user reaches its source. The index did not stop working; it became capable of completing the job that once belonged to the click.

A Wall Street Journal review concluded that AI search looked more like a user-experience revamp than a wholesale replacement for blue-link search. That is the strongest counterargument, and it does not reverse the structural change. A terminal can retain a directory inside it. What matters is whether navigation remains the default end of the interaction, not whether links remain visible somewhere on the screen.

The answer layer keeps the demand it once exported

The division of labor in search depended on Google exporting monetizable intent. A query about shoes, insurance, travel, or a medical symptom carried commercial or subscription value, but a publisher or merchant could capture that value only when the user arrived. Answer-first search compresses discovery and resolution into one interface.

Google’s plan to test search and shopping ads inside AI Overviews, drawing from advertisers’ existing campaigns, showed how little plumbing had to change for the answer layer to capture that intent. The advertisements could move upward into the generated response while publisher visits became less frequent.

Even some links embedded in AI Overview terms lead to another Google Search results page rather than directly to an outside publisher. Google says those links help people explore topics and discover relevant websites, and they can. They also add another internal step between the source and the visit. The same element can support discovery and retain attention; interface design decides which function dominates.

OpenAI was reportedly planning a browser that would keep some interactions inside a ChatGPT-like interface rather than sending users through to websites. Retention is becoming a category-wide design goal because an assistant that finishes a task controls more valuable demand than one that merely recommends where to finish it.

As prediction gets cheaper, companies compete over the assets prediction cannot supply by itself: trusted current information, interface defaults, advertising relationships, compute capacity, and the right to use source material. Search once created value by ordering destinations. The answer engine creates value by absorbing work once done at those destinations.

Vanishing referrals turn permission into inventory

Publishers tolerated broad crawling because referral traffic acted as payment in kind. The bargain could remain implicit while the crawler and publisher each received something the other controlled. As referrals decline, publishers gain leverage at the permission boundary: whether content may be crawled, retained, cited, used to ground an answer, or licensed for training and real-time retrieval.

A TollBit analysis of 160 websites using its content-licensing service found that AI search engines drove 96% less referral traffic than Google Search. Citation cannot settle that deficit by itself. A source name may provide attribution, but attribution is not a visit, and a visit is not automatically revenue. Each is a different economic instrument.

Microsoft made that distinction explicit when it launched a Publisher Content Marketplace with Condé Nast, Hearst, the Associated Press, and others. Publishers can set terms, track use, and receive payment when AI systems ground answers in licensed premium content. The marketplace turns an informal crawling convention into metered inventory.

CNN marked the same transition from the other side when it sued Perplexity for allegedly copying and distributing its content after the companies failed to agree on terms in 2025. Their failed negotiation matters more than the complaint: once traffic is no longer adequate compensation, access either acquires a price or becomes a dispute.

Not every publisher has equal leverage. Current, authoritative, or differentiated material is harder to substitute than generic pages assembled around common queries. A marketplace can aggregate rights, but it cannot manufacture scarcity for content that answer engines can obtain elsewhere. The open web is separating into priced inputs, replaceable inputs, and withheld inputs.

Retained attention comes with a meter attached

A list of links is cheap to serve compared with a generated response that must retrieve material, process tokens, synthesize an answer, and sometimes continue through several conversational turns. Once search retains the interaction, it also retains the recurring inference bill. The interface that captures more attention attaches a compute cost to each unit of it.

Barclays projected that inference capital expenditure would surpass training within two years and reach $208.2 billion in 2026. OpenAI engineers reportedly found a method that could more than halve inference costs, showing how much value sits inside small efficiency gains multiplied across repeated queries.

Sources say Google is developing a specialized server chip informally called Frozen v2, integrating Gemini’s model blueprint into silicon for a planned 2028 deployment. The program remains rumored, not a confirmed Google product plan. Such hardware could lower the cost of keeping users inside an interface where every retained query creates an inference expense.

The cloud abstraction ends in financed equipment. A study estimated that off-balance-sheet debt at Alphabet, Microsoft, Amazon, Meta, and Oracle grew roughly eightfold since 2022 to $1.65 trillion, tied to GPU contracts, data-center leases, and joint ventures. Those obligations sit beside the long-duration contracts securing powered compute capacity. Concrete buildings, electrical connections, accelerator racks, and lease payments support an interface whose most visible feature is that the user did not have to open another tab.

By keeping the interaction, the answer engine keeps the advertising opportunity and returns less traffic to publishers. It must also pay for inference and, increasingly, differentiated content. The old system exported the expensive last step to the destination site. The new system internalizes more of the value and more of the cost.

Regulation has followed the gate, not the page

Ranking rules governed the previous phase because placement on the results page determined who received traffic. In answer-first search, the contested assets sit earlier and deeper: the device default that receives the question, the interface that decides whether a click is needed, the search data that improves the system, and the source controls that determine which material enters the response.

The EU’s Digital Markets Act decisions require Google to give rival AI assistants and search engines comparable access to Android and some Search data. Regulators are treating assistant access and search data as competitive infrastructure. A rival model without distribution can remain invisible, while a rival interface without source access can remain uninformed.

Perplexity’s Computer feature splits tasks between local and cloud models to keep private data on-device and maximize token efficiency. Not every interaction must be centrally processed, and not every assistant must impose the same infrastructure bill. Competitors can challenge the gateway through defaults, data access, rights agreements, or the physical location where inference occurs.

Regulation can open Android access or require some Search data sharing. It cannot restore the old referral exchange once users have learned that a search page can finish the task. Publishers can demand licenses or withhold material, but they cannot turn a citation into the same commercial event as a visit. The web’s destinations have become suppliers to the systems that increasingly stand between them and their audience.

The 18-to-1 crawl-to-visit ratio is the receipt: Google’s blue link remains on the sign, but the load terminates in its data center, while the publisher waits at the service gate with a license.