Google says the vast majority of AI Overviews provide high-quality information and many of the viral examples have been uncommon queries or have been doctored

Step 1: Google rolls out a new AI-powered product. Step 2: Users quickly find the product's flaws and point them out with social-media posts, which become news stories.

Business Insider 2024-05-26 Peter Kafka

Context & Ripple Effects

The story captures Google’s initial public defense as criticism of AI-generated search answers spread through social posts. Related coverage shows the company was already removing certain AI Overview responses under its content policies, making reliability management—not merely product messaging—the immediate issue.

The subsequent explanation that Google added detection for nonsensical queries and limits around satire and humor shows how quickly a public-quality dispute can become a systems and policy update cycle for a widely distributed AI feature.

First-order effects

Google must defend confidence in AI Overviews while distinguishing broad product performance from highly visible edge-case failures and manipulated examples.
Users encountering questionable answers face a less predictable search experience while Google removes or limits responses for some queries.

Second-order effects

The episode raises the operating cost of AI search: Google has to add evaluation, query detection, and content-policy safeguards alongside model quality improvements.
Rival search and AI products gain a clear comparison point, while Google’s search audience has more reason to judge AI answers by reliability rather than novelty.

Third-order effects

If these incidents recur, generative search will be governed increasingly as a live service with continuous monitoring, selective abstention, and rapid policy intervention rather than a one-time model launch.
The durable competitive advantage may shift toward platforms that can safely deploy AI at search scale and make failures observable, correctable, and bounded.

The trend: AI search is moving from headline-grabbing feature launches toward operational governance of model outputs in high-frequency consumer workflows.

Discussion

@jbillinson Josh Billinson on threads
Google should obviously not be telling people to make pizza with Elmer's glue. Everyone knows the secret to a good pizza is using New York tap glue.
@andrewwebster Andrew Webster on threads
cool how google turned search into the worst system shock sequel
@fredericl Frederic Lardinois on threads
I'm trying to get worked up about the mistakes Google's AI Overview makes, but I'm failing. In day-to-day usage, the examples I see are alright. I'm more worried that it's actually quite good (or at least good enough and getting better) and takes away clicks from publishers.
@bzamayo Benjamin Mayo on threads
Google AI overviews aren't in my region yet I'm missing the opportunity to engagement farm off its stupidity
@alexhcranz Alex Cranz on threads
companies continue to poison any goodwill they have by pushing out products that aggressively lie to you by also (possibly) stealing all your content to lie to other people. the companies don't see it that way—but the general public sure as shit is
@technicallymims Christopher Mims on threads
the fact that nearly all of us into believing that Google's generative search results told people to jump off a bridge is a near-perfect illustration of the fact that AI has *already* confused us into not knowing what is true anymore, just by the fact if its existence
@quillmatiq Anuj Ahooja on threads
Microsoft wanted Google to dance but it turns out Google has two left feet tied at the ankles 💃🏼
@drewharwell Drew Harwell on threads
If there's one thing I know about Google, it's that people always search for normal stuff
@kylie.robison Kylie Robison on threads
Google confirms it manually removes odd AI search results, and I discussed the company's super-smart-search ambitions with Gary Marcus (who remembered a time when a Meta AI model told people to eat glass) https://www.theverge.com/...
@alex.stamos Alex Stamos on threads
I strongly recommend not amplifying any claims about Google's AI output without trying the query yourself, as lots of fake screenshots are circulating. Even The NY Times fell for it:
@wongmjane Jane Manchun Wong on threads
another crazy Google AI Overview result lol
@mhwilliams.bsky.social M.H. Williams on bluesky
Which is all well and good as a statement, but users can actually do the queries themselves and it's quite clear they still have the same problems that all LLMs do. The hallucinations are rampant and you're not really going to be able to get rid of them at scale. [embedded post…
@m33uhh.bsky.social Mia on bluesky
I thought one of the main benefits of AI generated results was being able to ask incredibly specific questions, aka “uncommon queries”
@dell.bsky.social Dell Cameron on bluesky
The translation is that Google knows it is supplying misinformation & that it has an acceptable failure rate. — What is a “vast majority”? Is it 80%? 90%? How many tens of millions of results does that leave? This is an intentional design flaw & when harm results it should …
@mattvallone.bsky.social Matt Vallone on bluesky
It's been odd watching them transition to “Do be evil”. [embedded post]
@monospace.com Hens Breet on bluesky
The problem is that we don't know which is which so it's safest to assume it's all compromised.
@ernie.tedium.co Ernie Smith on bluesky
More generally, I want an unadulterated experience when I look at Google. The level of visual noise has been rising, but this particular feature pushes it over the edge. [embedded post]
@ablum.bsky.social Andrew Blum on bluesky
For me, it's not just the inaccuracies. The viral examples we've seen remove any and all mystique AI has. I mean if you trace the mistake to a single Onion article... [embedded post]
@douglasmack.bsky.social Doug Mack on bluesky
The issue isn't that AI is sometimes wrong (per se), the issue is that all info is presented with the ILLUSION of expertise but stripped of all context or attention to details like “is this source reliable or is it complete bullshit that will end up killing me in five different w…
@brendansinclair.bsky.social Brendan Sinclair on bluesky
Oh, well that's ok if it's just edge cases then. If Google's AI only tells you to do incredibly dangerous things in .01% of cases, and Google handles 3 billion searches a day, that's only like 300,000 people a day being told to drink bleach or whatever. Seems fine. [embedded p…
@ernie.tedium.co Ernie Smith on bluesky
Did they say anything about &udm=14? [embedded post]
@grudgie.bsky.social @grudgie.bsky.social on bluesky
“Most of our information is good” simply is a generous way of saying “you cannot rely on our information.” [embedded post]
@calizaybak.bsky.social @calizaybak.bsky.social on bluesky
Who wants a search tool that can handle “uncommon queries”? — And doctored! Except all the examples I've seen here have a comment that says “OMG, I tried it myself!” which is a thing anyone can instantly do so doctored examples would get dunked-down instantly. [embedded post]
@cdarklock.bsky.social @cdarklock.bsky.social on bluesky
Uncommon queries are YOUR FUCKING JOB — Nobody is asking your search engine about things ANY JACKASS ALREADY KNOWS — What the entire fuck [embedded post]
@shevek.bsky.social @shevek.bsky.social on bluesky
you see this is fine and good because the big query machine doesn't *always* tell you formaldehyde is a great decongestant [embedded post]
@joshgrant.online Josh Grant on bluesky
Also, basically all of their customers of Google Cloud in Australia haven't seen any problems! [embedded post]
@braak.bsky.social Braak on bluesky
Google at long last providing the people what they want, answer to their questions that are mostly true [embedded post]
@beijingpalmer.bsky.social @beijingpalmer.bsky.social on bluesky
‘by 2024 Baidu will be better at search than Google.’ — 'wow! So Baidu's really going to improve?' — 'I didn't say that' [embedded post]
@zitron.bsky.social Ed Zitron on bluesky
oh okay no problem thank you!!!!!!!!! [embedded post]
@pkafka Peter Kafka on x
Oh and here's Google's newest statement re: Ai search. Full speed ahead! [image]
@mimbsy Caroline Mimbs Nyce on x
Google's new AI Overview told me that the one-year survival rate for chemotherapy is 52 percent. That statistic is from a real scientific paper about head and neck cancer, which it mistakenly applied it to *all* cancers. [image]
@pkafka Peter Kafka on x
I'm old enough to remember when Google bailed on an ai product after public backlash. In February. But I think they're fully locked into this one: gripes are one thing but backing down would be a disaster for management. Speaking of disaster: “let's discourage people from usin…
@ben8128 Ben Eidelson on x
The bad results are sometimes funny and, yes, embarrassing to Google. And I also find AI overviews to be a tremendously useful feature. Two things can be true at once. Especially for a beta product.
@willoremus Will Oremus on x
“With the generative AI from Google, OpenAI's ChatGPT and Microsoft's Copilot, you should assume that they're wrong until proved otherwise.” — @ShiraOvide on how to navigate a world in which Google tells you to eat rocks and glue https://www.washingtonpost.com/ ...
@chaykak Kyle Chayka on x
it's hard to overemphasize this: Google and OpenAI have no plan for how or why people will generate *new, correct information* in the age of generative AI search
@jjvincent James Vincent on x
it's completely damning of google's principles that its defense for the failure of AI search is that the queries being asked are “uncommon.” like a library telling you the more infrequently a book is taken out, the less you should trust it https://www.theverge.com/... [image]
@tomwarren Tom Warren on x
I know everyone is laughing at Google for its AI search results suggesting we should eat rocks. The funniest part is that the rocks thing also appeared on regular featured snippets, so it's not just AI results. Perhaps Google is more broken than we think [image]
@ethank Ethan Kaplan on threads
“Apple is late to AI”
@paul_rietschka Paul Rietschka on threads
“Google inadvertently reinvents expert systems.” This will end well.
@jonathangarelick Jonathan Garelick on threads
Google's generative AI team is writing a crazy amount of if statements right now
@reckless1280 Nilay Patel on threads
Google is manually removing weird AI answers in search - probably ruining whatever margin bringing the cost of an AI query down 80 percent over the last year saved. Lots in this @kylie.robison piece! https://www.theverge.com/...
@kylie.robison Kylie Robison on threads
Google promised a better search experience — now it's telling us to put glue on our pizza Last year, Google said that AI was the future of search. What's the point, though, if the search seems dumber than before? https://www.theverge.com/...
@laguettler.bsky.social @laguettler.bsky.social on bluesky
Thank goodness the company that created a dangerously unreliable product will be determining in how much danger is worth the financial and reputational investment they sunk into it. [embedded post]
@darthbluesky.bsky.social @darthbluesky.bsky.social on bluesky
will an AI be making these decisions over what will be removed [embedded post]
@darthbluesky.bsky.social @darthbluesky.bsky.social on bluesky
how dare users try to trip up our experimental AI that we are training on these same users to make a lot of money for ourselves where is the loyalty
@benedictevans Benedict Evans on threads
I am hearing disturbing rumours that if you do weird random searches, occasionally you get weird random results
@moskov Dustin Moskovitz on threads
Many people are using the Google search AI examples to declare LLMs a dead end, but there are other search products today, including Perplexity and Bing, that do not have these problems. Google can figure out how to frame Reddit answers appropriately and exclude theonion from fa…
@davemark@mastodon.social Dave Mark on mastodon
Rollup of some of the wildly incorrect answers Google's AI search is surfacing. — Garbage in. Garbage out. 😐 — https://www.theverge.com/... #AI #Google #Futurology
@rakeshlobster Rakesh Agrawal on x
They brought this on themselves. They could easily have limited it to 1% of their users. Speculation: they're rolling it out across everyone before DoJ considers it illegal tying - using their market dominance in search to jumpstart their AI product. If I were an antitrust
@edzitron Ed Zitron on x
I can't get over this interview from The Verge with Sundar Pichai - he made Google integrate generative AI into search despite hallucinations being “unsolved,” because it makes them “very creative.” I don't want Google to be creative!!! I NEED AN ANSWER!!! https://www.theverge.co…
@swilua @swilua on x
the thing is, this is based on a real peer reviewed study! It also happens to be a satirical exploration of the limits of randomized trials if you trust AI to do your thinking for you, it's going to miss some pretty big, pretty important things [image]
@random_walker Arvind Narayanan on x
Some of these screenshots are likely fake but go viral because people are no longer skeptical that Google could be this broken. Given the non-deterministic and transient nature of AI responses there's no way to know for sure. 25 years of trust flushed down the toilet.
r/skeptic r on reddit
Google promised a better search experience — now it's telling us to put glue on our pizza - The Verge

Chronicles