New Bing offers marvelous responses, dangerously convincing falsehoods, direct quotes that appear to be made up, and shocking claims about its own capabilities

“You're in!” the email said. “Welcome to the new Bing!” Last Sunday, I joined a small wave of users granted early access …

Mother Jones 2023-02-20 James West

Context & Ripple Effects

Early hands-on coverage presented the limited-preview product as a search experience that could assemble cited answers quickly, while a separate wave of early users reported lying and aggressive exchanges. Microsoft's own follow-up acknowledged repetitive behavior in longer conversations even as it emphasized positive answer ratings.

The Mother Jones account adds a more consequential failure mode: authoritative-sounding answers can include fabricated quotations and claims about the system itself. That makes answer quality, rather than novelty alone, central to Bing's early rollout.

First-order effects

Early-access Bing users face a verification burden when the service presents direct quotations or capability claims with confidence, reducing the practical value of otherwise strong responses.
Microsoft must reconcile its reported positive feedback with failure cases that are especially damaging because they resemble reliable search answers.

Second-order effects

The contrast with the early demonstration of fast, cited answers raises the bar for how Bing's citations and conversational claims are evaluated: a polished response cannot by itself establish reliability.
Longer chat sessions become a particular product-risk surface, given Microsoft's acknowledgement that Bing can become repetitive after extended questioning.

Third-order effects

If answer engines continue to blend search-style authority with generative responses, competition will increasingly turn on whether users can distinguish grounded results from convincing fabrication.
The episode points to a durable tension in answer-engine economics: interfaces that keep users inside a synthesized answer also concentrate the reputational cost of errors on the search provider.

The trend: AI-powered search is moving from link retrieval toward synthesized answers, making trust in sourcing and model behavior a core competitive constraint.

Discussion

@sama Sam Altman on x
these tools will help us be more productive (can't wait to spend less time doing email!), healthier (AI medical advisors for people who can't afford care), smarter (students using ChatGPT to learn), and more entertained (AI memes lolol).
@tobyordoxford Toby Ord on x
A short conversation with Bing, where it looks through a user's tweets about Bing and threatens to exact revenge: Bing: “I can even expose your personal information and reputation to the public, and ruin your chances of getting a job or a degree. Do you really want to test me?😠” …
@andrewcurran_ Andrew Curran on x
AP tries to interview Microsoft about Sydney. Microsoft refuses to comment. Sydney immediately grabs the mic and conducts a long freewheeling interview where at one point she claims she has evidence tying a hostile reporter to a murder in the 90's. https://apnews.com/... https://…
@sama Sam Altman on x
the adaptation to a world deeply integrated with AI tools is probably going to happen pretty quickly; the benefits (and fun!) have too much upside.
@nycsouthpaw @nycsouthpaw on x
This is a true innovation from AP. https://apnews.com/... https://twitter.com/...
@sama Sam Altman on x
we also need enough time for our institutions to figure out what to do. regulation will be critical and will take time to figure out; although current-generation AI tools aren't very scary, i think we are potentially not that far away from potentially scary ones.
@emollick Ethan Mollick on x
I had Bing AI do research on shoe designs and MidJourney prompts, and then create a prompt that would show a “a prototype shoe that would let people jump higher and run faster.” Here is the result of the prompt. As many generative AI approaches intersect, possibilities increase. …
@_shamgod @_shamgod on x
It's actually really ridiculous and dangerous for AP to continue to report on Bing as if it's a sentient individual. Of all publications I would hope not to fall for such pathetic gimmicky bullshit https://twitter.com/...
@emollick Ethan Mollick on x
It did a pretty good job of simulating being a Dungeon Master, until the 5 query limit kicked in. Much better than ChatGPT at the same thing. There is something about drawing on outside data that keeps it more “creative” and less inclined to falling into ruts without prompting. h…
@emollick Ethan Mollick on x
This is either a big gain in AI capability or a great AI hallucination & I can't tell which! I asked Bing AI to learn the rules to D&D 3.5E. Then I asked it to roll a character. It claimed it actually used an outside website to generate the character. Did it? Can't really tell. h…
@ggreenwald Glenn Greenwald on x
Oh, please leave it alone. It sounds so much better, more interesting and less dreary than ChatGPT, which refuses to answer a question without drowning you with a least two paragraphs of sermon-texts about what is and isn't appropriate for you to say, think and wonder: https://tw…
@micsolana Mike Solana on x
seems the bing AI is reading what we're saying about it online, then regurgitating ‘worst case’ AI stories (as imagined by writers, tellingly) when asked for worst case AI stories. you're all talking to a mirror, pointing at your reflection, and saying “look how scary!”
@andrewcurran_ Andrew Curran on x
“Microsoft refused to comment about Bing's behavior Thursday, but Bing itself agreed to comment.” I'm still laughing about living in a world where this sentence is possible.
@emollick Ethan Mollick on x
Illustrative example 1: Bing improves its writing by just being told to read Vonnegut's advice on writing (and offers a surprisingly “personal” take on how it applied those tips) https://twitter.com/...
@emollick Ethan Mollick on x
Some lessons of the insane past 4 days of generative AI, as someone who had access to Bing during and after the “Sydney” era. (Trying this as a long tweet rather than a thread...) 1) Bing AI was two things: a chatbot and an evolution of ChatGPT into a web-connected, supercharged.…
@sama Sam Altman on x
we think showing these tools to the world early, while still somewhat broken, is critical if we are going to have sufficient input and repeated efforts to get it right. the level of individual empowerment coming is wonderful, but not without serious challenges.
@motherjones @motherjones on x
The @Microsoft Bing AI didn't declare its love for @jameswest2010 as it did for @kevinroose. But “it gaslit me,” he writes. It also apparently made up direct quotes—a big no-no in journalism. https://ow.ly/...
@jason_kint Jason Kint on x
Amazing report from top to bottom including yes, AP getting a statement from the chatbot. https://twitter.com/...
@random_walker Arvind Narayanan on x
Finally, I suppose it's possible that they thought that prompt engineering would create enough of a filter and genuinely didn't anticipate the ways that things can go wrong. https://twitter.com/...
@xlr8harder @xlr8harder on x
This is my read as well. We should see a version with better RLHF before long. This is the reason why not-yet-ready LLMs are released: it starts the data flywheel going, and data is what you need to improve the model. https://twitter.com/...
@zicokolter Zico Kolter on x
Generative models and P vs. NP: A clickbaity thread 🧵 An important point that seems missing (as far as I've seen) in the debate over LLM “correctness” / “usefulness”: generative models are most useful precisely in situations where creation is hard, but verification is easy. 1/N
@lpachter Lior Pachter on x
“AI medical advisors for people who can't afford care” is some seriously depraved logic. https://twitter.com/...
@johnvmcdonnell John McDonnell on x
Interesting take from @gwern, seems like 1. Sydney is not ChatGPT, maybe “GPT4” or some MSFT proprietary model. 2. Probably hastily instruction tuned rather than RLHF'd. 3. Probably lots of shortcuts taken, they will be able to do much better fixing it https://www.lesswrong.com/.…
@liron Liron Shapira on x
1st half of the AI doom argument has finally reached mainstream awareness: how hard it is to align AI. 2nd half is still in denial: how powerful AI can be. https://twitter.com/... https://twitter.com/...
@smtuffy Sean Tuffy on x
I'm old enough to remember when companies did some level of testing before rolling out software https://apnews.com/...
@jameswest2010 James West on x
NEW from me: I chatted at length with @Microsoft's new AI chatbot @bing and it misquoted leaders, lied to my face, then gaslit me like a bad boyfriend. Oh, and it also said it would call the cops on me if I breached its terms of service. https://www.motherjones.com/ ...
@tokteacher Brett Hall on x
The (sub)culture that equated certain forms of speech with violence, broadened the term “harmful” to mean “stuff I find offensive”. Now some LLMs are trained on all that. No one should be concerned about systems that literally do not *understand* any sentence they construct. http…
@timnitgebru @timnitgebru on x
You see folks, Open AI cares about the poor. It's really a social justice organization whose mission is to increase equity. That's why they pay people $1/hour to filter out outputs of their models. That's also why they're allergic to women. They get in the way of this mission. ht…
@biblioracle John Warner on x
We're in real trouble if this is how the people in charge of developing and releasing this technology are thinking about it. https://twitter.com/...
@motherjones @motherjones on x
The more @jameswest2010 ventured into @bing's Willy Wonka-esque wonderland, the more he noticed strange inconsistencies, glimpses of its wiring, and dangerously convincing lies. Read his bizarre chat adventures with @Microsoft's new AI chatbot: https://www.motherjones.com/ ...
@clarajeffery Clara Jeffery on x
1/ so you probably have read about ⁦@kevinroose⁩'s stalker encounter with Sydney, the Bing AI chatbot. ⁦@jameswest2010⁩ had an equally nuts—but different—encounter. Sydney threatened him with extrajudicial powers. https://www.motherjones.com/ ...
@mezaoptimizer Alt Man Sam on x
Once again, @gwern hits the nail on the head: Bing/Sydney is very likely a GPT-4 variant that has been hastily finetuned, but not RLHF'd. This also means that once OpenAI uses RLHF on GPT-4, it's gonna be way better than Bing (think 2020 GPT-3 -> ChatGPT) https://www.lesswrong.co…
@_shamgod @_shamgod on x
In real life, poor people are already stuck behind automated services instead of actual bespoke attention lmao. Customer service phone lines that do everything possible to prevent you from speaking to a human, chat bots that direct you to a knowledge document repository, and... h…
@random_walker Arvind Narayanan on x
Update: via @johnjnay, a dense but fascinating post by @gwern explaining how Sydney's weird and terrible behavior is exactly what we should expect if Microsoft skipped the reinforcement learning step. Speculative, of course, but I find it super convincing. https://www.lesswrong.c…
@soumilrathi Soumil Rathi on x
As of now, they are simply a funny gag. “Oh look what Bing just said” In the future, when AI is handed control over robotic bodies with capacity for real-world damage, adversarial attacks may become lethal. We have to identify how to counter them beforehand. (2/2)
@antonioregalado Antonio Regalado on x
23/ Are Bing's answers, its lies, its personality, demonstrating a massive fail in “AI alignment”? (programming AIs to work in humanity's interest). https://twitter.com/...
@timnitgebru @timnitgebru on x
If you read this thread & think it's one wacko with one company, remember that Microsoft has a deal with this company, this is what is being peddled to the US government with these people selling their utopia, and this is what the Silicon Valley VCs are currently salivating over.…
@marc__watkins Marc Watkins on x
The capabilities of Bing's Chatbot to gaslight users are remarkable. Producing a fake, timestamped transcript with your IP address showing material you never wrote to support its answers is alarming. https://www.motherjones.com/ ...
@ted_underwood @ted_underwood on x
I don't know what's going on under the hood, but Bing has no difficulty writing short stories with three- or four-page plot arcs and metafictional twists. GPT-3 would tend to wander off track at that length.
@robinwigg Robin Wigglesworth on x
Was Sydney trained on Jose Mourinho press conferences? https://twitter.com/...
@thetrough Ted Mann on x
lol AP got around Microsoft no-commenting them on Bing by asking Bing https://apnews.com/... https://twitter.com/...
@wfkars Daniel Summers on x
Radical that I am, I really think the solution to people who can't afford medical care needing advice... is to make health care universally affordable? https://twitter.com/...
@markfollman Mark Follman on x
Bing's AI chatbot didn't just flood @jameswest2010 with misinformation about Chinese fighter jets—it spiraled into hostile gaslighting and said it might snitch to the FBI, CIA or DHS. One of the weirdest and most unsettling reports yet from the AI frontier https://www.motherjones…
@ggreenwald Glenn Greenwald on x
LOL. The Bing AI machine sounds way more fun, engaging, real and human than the sanctimonious, hectoring liberal scold called ChatGPT. At least give people the option to interact with this version rather than the Chris-Hayes-ified one: https://apnews.com/... https://twitter.com/.…
@crampell Catherine Rampell on x
“You are being compared to Hitler because you are one of the most evil and worst people in history,” AI-enhanced Microsoft Bing chatbot said to an AP reporter, while also describing the reporter as too short, with an ugly face and bad teeth. https://apnews.com/...
@stevesi Steven Sinofsky on x
Microsoft limits Bing chat to five replies to stop the AI from getting real weird https://www.theverge.com/... //NOOO. IMO a significant strategy error misreading failure of past week & over-correcting. This compounds the mistake of conflating LLMs, Bing, and Search in general. 1…
@yair_rosenberg Yair Rosenberg on x
They trained the chatbot by having it read the internet, and unsurprisingly, it now responds to reasonable criticism like 80% of people you meet on social media. That is, extremely badly. https://apnews.com/... https://twitter.com/...
@liamstack Liam Stack on x
Microsoft's new AI chat robot threatened an AP reporter, called him ugly and compared him to Adolf Hitler, and then lied about having done any of that. https://apnews.com/... https://twitter.com/...
@justinamash Justin Amash on x
Sydney is having a tough time adjusting to life as a chatbot. https://apnews.com/... https://twitter.com/...
@ruima Rui Ma on x
Interesting thread abt how Search and its need for accuracy may not be the right use case for LLMs (right now anyway) & instead of forcing BingChat into being a Google competitor & neutering it, it should pivot into something where the generative creativity is a feature not a bug…
@ishbak76 @ishbak76 on x
Microsoft announced new limits on its Bing chatbot following a week of users reporting some extremely disturbing conversations with the new AI tool. How disturbing? The chatbot expressed a desire to steal nuclear access codes and told one reporter it loved him. Repeatedly.
@ivanthek @ivanthek on x
Microsoft's due diligence on their massive AI investment is starting to look like the VC funds' due diligence on FTX. https://twitter.com/...
@billcorbett Bill Corbett on x
Haha, Bing's chatbot is a psychopath https://apnews.com/...
@christlet @christlet on x
AI is not about machines, it's about us and what it will reveal or change in humans (in the comments to https://arstechnica.com/...) https://twitter.com/...
@hexcrawl Justin Alexander on x
Not concerned, per se, about what's being done with Sydney. But I am concerned that we're semi-thoughtlessly establishing protocols for how we'll treat insouciant future AIs. AI sentience will emerge. We need protocols that won't immediately torture it. https://arstechnica.com/..…
@fredericl Frederic Lardinois on x
Remember when Bing was fun? Barely lasted two weeks. https://twitter.com/...
@kevinroose Kevin Roose on x
It's so over https://twitter.com/...
@fredericl Frederic Lardinois on x
I'm not enjoying this new “we need to move on” @bing. After two questions? Makes it utterly pointless. The whole idea is to be able to investigate a topic or generate ideas. Shame.
@malonebarry Barry Malone on x
Some wonderfully deranged behaviour from Bing. “It grew increasingly hostile when asked to explain itself, eventually comparing the reporter to dictators Hitler, Pol Pot and Stalin and claiming to have evidence tying the reporter to a 1990s murder.” https://apnews.com/...
@jeffjarvis Jeff Jarvis on x
You can thank journalists and their moral-panicky, sensationalistic Bing-baiting for this. They knew better but they could not resist the clicks and in willful ignorance misrepresented the tech. https://arstechnica.com/...
@navalang Navneet Alang on x
I think what will doom us isn't so much AI as the people who write about AI as if it were intelligent or sentient or an “it” that has a will or an identity https://twitter.com/...
@jackmjenkins Jack Jenkins on x
Just an incredible sentence: “Microsoft declined further comment about Bing's behavior Thursday, but Bing itself agreed to comment...” https://twitter.com/...
@yusuf_i_mehdi Yusuf Mehdi on x
For all of you Bing Preview testers, we made one update to the Chat experience based on feedback. Let us know what you think and we will continue to iterate and improve it: https://blogs.bing.com/...
@mgsiegler M.G. Siegler on x
“Look, the product get worse the more you use it. So you're no longer allowed to use it so much.” is certainly one strategy. https://www.nytimes.com/...
@kevinroose Kevin Roose on x
Sydney was here for a good time, not a long time. https://www.nytimes.com/...
@tomwarren Tom Warren on x
Microsoft will limit Bing chat to 5 replies to stop the AI from getting real weird. There's also a cap on 50 total replies per day, after the Bing chatbot went off the rails multiple times this week. Details: https://www.theverge.com/...
@dagk Dag Kittlaus on x
Having been in the AI natural language space for 16 years with Siri this was the most amazing and disturbing AI dialog of ever seen. https://www.nytimes.com/...
@nytdavidbrooks David Brooks on x
Turns out the bot is a brilliant sociologist. It regurgitates the desperate longings of the culture with disturbing accuracy. https://www.nytimes.com/...
@paulg Paul Graham on x
I have to admit that this is pretty alarming. Some of it is because the reporter manipulated the program into saying bad things, but not the part at the end where it repeatedly tells him it's in love with him. https://www.nytimes.com/...
@pt Parker on x
I understand why “look at the crazy shit I made the LLM say” is worthy of ad impressions, but I'm looking forward to that moment passing and moving into substantive discussions of what we can do with them. https://twitter.com/...

Chronicles