Users say Google's Gemini Pro is loath to comment on potentially controversial news topics, fails to get basic facts right, and struggles with some coding tasks
TechCrunchKyle Wiggers
Context & Ripple Effects
This report captures an early trust test for Gemini Pro: users were evaluating not only whether the model could answer difficult prompts, but whether its answers and coding help were dependable enough for routine use.
Users encountering the reported refusals, factual errors, and coding failures have less reason to rely on Gemini Pro for news-adjacent research or programming tasks without independent checks.
Google faces an immediate quality-control problem: improving factual reliability, task performance, and guardrail behavior at the same time rather than treating them as separate product issues.
Second-order effects
The reports raise the value of comparative testing among AI assistants, especially for users whose workflows depend on current-information queries or usable code output.
Repeated fixes and capability work can slow feature delivery as Google allocates more evaluation and engineering effort to failures exposed in public use.
Third-order effects
If this pattern persists, frontier AI competition will be judged increasingly on dependable behavior across edge cases, not just headline model capability.
The episode illustrates the lasting tension between safety restrictions and usefulness: systems that avoid risky outputs can still lose trust when their boundaries appear inconsistent or their answers unreliable.
The trend: Consumer AI assistants are moving from novelty products to reliability-sensitive tools, making evaluation, guardrails, and task-specific performance central competitive variables.
Somewhere a Google engineer notices that the data center numbers are redlining as the AI tries to figure this out
@vitor_dlucca
@vitor_dlucca on x
I'm extremely disappointed with Gemini Pro on Bard. It still give very, very bad results to questions that shouldn't be hard anymore with RAG. A simple question like this with a simple answer like this, and it still got it WRONG. [image]
@filippie509
@filippie509 on x
Tried gemini based Bard, and well, it still can't write intersection of two polygons. It's one of those rare relatively simple to express functions that wasn't ever implemented in python, there is no stack overflow post, and all these models fail on it. [image]
@benjaminnetter
Benjamin Netter on x
FYI, Google Gemini is complete trash. [image]
@mohsen____
Mohsen Azimi on x
Trying out Gemini Pro: it is pretty disappointing for my example. I asked it to make an analog clock using HTML like this one that ChatGPT made. It can cite some code from Github but it's off by a few ms... [video]
@buzzedison
Edison Ade on x
GPT 4 still greater than Gemini Pro. Created Tic Tac Toe game with ChatGPT and Bard(Running on Gemini Pro) See video for the result. ChatGPT wrote the code on first try(First Video). Bard on 3 tries(Second Video). [video]
@emollick
Ethan Mollick on x
We really don't know anything about Gemini Ultra. Does it beat GPT-4 for real? If so, why by such a small amount? Two options: 1) Gemini represents the best effort by Google, and the failure to crush GPT-4 shows limits of LLMs approaching 2) Google's goal was just to beat GPT-4
@tomgara
Tom Gara on x
This is wild, the car one in the demo seemed super impressive for this exact reason
@charlesarthur
Charles Arthur on x
Even at the edited speed I wasn't impressed. Should I have been?
@tunguz
Bojan Tunguz on x
For comparison, here are the answers I got from Grok, ChatGPT, and Bing: [image]
@modestproposal1
@modestproposal1 on x
so basically AGI
@techcharts
@techcharts on x
Hey Gemini, tell me what you are seeing? Gemini: I see a cup & handle chart pattern that can act as a continuation. The handle is forming above the 200-day average, so if there is a breakout it will be in the direction of the uptrend. Trending in the direction of the existing... …
@ztobi
Tobias Zwingmann on x
@tunguz At the bare minimum it should have shown a link to let me google that for you.
@filippie509
@filippie509 on x
What it does, is it calculates the list of points where two edges of input polygons intersect, but that obviously is not the same as the polygon that is an intersection of the two.
@buzzedison
Edison Ade on x
[video]
@vitor_dlucca
@vitor_dlucca on x
Meanwhile, Bing gave a COMPLETE list of winners. And all of them are right (as far I know). [image]
@nkirankumars1
N Kiran Kumar on x
Just tested Google's Bard with Gemini Pro update. No bugless snake game on 1st try; reported, asked to fix—couldn't. Tried ChatGPT 3.5 free version, got correct bug-free code on the first attempt! 🚀🐍 #ChatGPT #Bard #Gemini [image]
@minchoi
Min Choi on x
@tunguz 🤔 [image]
@tunguz
Bojan Tunguz on x
So I just asked Gemini Pro in Bard for the updates regarding the hottest news story these days, the war in Israel and Gaza, and its answer essentially was, I kid you not, “just Google it.” 🤦♂️ [image]