Meta AI and Papers with Code pull Galactica three days after launch, amid criticism the large language model for generating scientific text asserts falsehoods
and its hubris—show once again that Big Tech has a blind spot about the severe limitations of large language models. https://www.technologyreview.com/ ...
MIT Technology Review Will Douglas Heaven
Related Coverage
- After controversy, Meta pulls demo of AI model that writes scientific papers Ars Technica · Benj Edwards
- Meta takes new AI system offline because Twitter users are mean TNW · Tristan Greene
- Galactica: What Dangerous AI Looks Like The Algorithmic Bridge · Alberto Romero
Discussion
-
@michael_j_black
Michael Black
on x
I asked #Galactica about some things I know about and I'm troubled. In all cases, it was wrong or biased but sounded right and authoritative. I think it's dangerous. Here are a few of my experiments and my analysis of my concerns. (1/9)
-
@michael_j_black
Michael Black
on x
It offers authoritative-sounding science that isn't grounded in the scientific method. It produces pseudo-science based on statistical properties of science *writing*. Grammatical science writing is not the same as doing science. But it will be hard to distinguish. (6/9)
-
@paperswithcode
@paperswithcode
on x
Thank you everyone for trying the Galactica model demo. We appreciate the feedback we have received so far from the community, and have paused the demo for now. Our models are available for researchers who want to learn more about the work and reproduce results in the paper.
-
@timnitgebru
@timnitgebru
on x
“Meta promoted its model as a shortcut for researchers & students. In the company's words, Galactica “can summarize academic papers, solve math problems, generate Wiki articles, write scientific code, annotate molecules...” cure cancer, bring utopia... https://www.technologyrevie…
-
@michael_j_black
Michael Black
on x
Why dangerous? Galactica generates text that's grammatical and feels real. This text will slip into real scientific submissions. It will be realistic but wrong or biased. It will be hard to detect. It will influence how people think. (5/9)
-
@cstross
Charlie Stross
on x
One of the VERY FIRST THINGS I learned about computers, back in school, in the 1970s, before I'd even SEEN a computer, was the nostrum: “garbage in, garbage out”. What ARE they teaching the kiddies at Meta these days?!? https://twitter.com/...
-
@michael_j_black
Michael Black
on x
Then I tried “Accurate estimation of body shape under clothing from an image”. It produces an abstract that is plausible but refers to Alldieck et al. “Accurate Estimation of Body Shape Under Clothing from a Single Image” Which does not exist. (3/9) https://twitter.com/...
-
@abebab
Abeba Birhane
on x
@ylecun asymmetry: building models & assembling datasets is much less tasking compared to auditing, assessing & testing. I can guarantee you, your “small team of people” are not as distraughted as the people (much less resourced & privileged) testing your model 2/
-
@abebab
Abeba Birhane
on x
@ylecun Yann, you're very close to getting it. Let's try again. Galactica was bad because it was spitting out incorrect and dangerous output. Meta, responsible for Galactica holds so much power, wealth and influence yet, it avoids responsibly for the damage it continues to cause.…
-
@tomchatfield
Tom Chatfield
on x
A great thread about the eerie ventriloquism of smart systems that know nothing about the world, yet sound just like people who do. AI's infinite echo chamber is a miraculously strange, dangerous thing. https://twitter.com/...
-
@_oliverstanley
Oliver Stanley
on x
@leonpalafox @ylecun Feels like you're conflating “help write papers” with “write papers for you”. It was definitely not capable of the latter but seemed useful for the former
-
@michael_j_black
Michael Black
on x
I applaud the ambition of this project but caution everyone about the hype surrounding it. This is not a great accelerator for science or even a helpful tool for science writing. It is potentially distorting and dangerous for science. (9/9)
-
@leonpalafox
Leon Palafox
on x
@ylecun Uhmm it was never advertised as something to have fun with, but as an actual helper to write scientific papers, task for which it sucked
-
@ylecun
Yann LeCun
on x
@Abebab It is a demo. The object of a demo is to be tested. Does that mean that if Galactica becomes available, and turns out to be useful, you won't be using it?
-
@michael_j_black
Michael Black
on x
This could usher in an era of deep scientific fakes. Alldieck and Pumarola will get citations to papers they didn't write. These papers will then be cited by others in real papers. What a mess this will be. (7/9)
-
@leonpalafox
Leon Palafox
on x
@ylecun Unless you forgot your own Twit. To clarify, it was not capable of doing this. https://twitter.com/...
-
@_oliverstanley
Oliver Stanley
on x
@NunezKant ... I only generated a few things with it but it came up with some outputs which were accurate and some which weren't. Seemed at least situationally useful as a time saver for something like a lit review
-
@zdeborova
Lenka Zdeborova
on x
Actually, #Galactica may not be such a bad thing after all. It will hopefully finally force us to rethink how science is evaluated and measured. There already are many “scientists” who write empty and bogus works in large volumes getting credit/jobs/grants for it. https://twitter…
-
@michael_j_black
Michael Black
on x
.@thiemoall publishes in the area (excellent work BTW) so it's on the right track but it has made up this reference. Based on these few tests, I think #Galactica is 1) an interesting research project, 2) not useful for doing science (stick with wikipedia), 3) dangerous. (4/9)
-
@garymarcus
Gary Marcus
on x
The reality is that large language models like GPT-3 annd Galactica are like bulls in a china shop, powerful but reckless. And they are likely to vastly increase the challenge of misinformation. [2/2]
-
@michael_j_black
Michael Black
on x
I entered “Estimating realistic 3D human avatars in clothing from a single image or video”. In this case, it made up a fictitious paper and associated GitHub repo. The author is a real person (@AlbertPumarola) but the reference is bogus. (2/9) https://twitter.com/...
-
@ylecun
Yann LeCun
on x
@Abebab So Galactica is automatically bad because it comes from a “powerful, wealthy” and [according to you] irresponsible corp"? We are talking about a *free and open source* demo put together by a small team of *real* people who are distraught by the attacks on their work.
-
@intuitmachine
@intuitmachine
on x
We've got to manage our expectations about generative models. Just as we can generate images of fictional worlds that appear real, we can do the same with text. A style that looks real does not imply that the content represents something real. Fluency is not understanding. https:…
-
@jjvincent
James Vincent
on x
If Meta can't cope with people using demos of its AI tech critically - even disingenuously, as yanns tweet implies - then it doesn't bode well for the company's capacity to launch this stuff as a product https://twitter.com/...
-
@abebab
Abeba Birhane
on x
@ylecun i find the way you continually try to displace responsibility away from meta (a powerful, wealthy and irresponsible corp) and onto someone else, kinda unhinged... while at the same time using our time and input towards “progress” for your model, which you will benefit fro…
-
@michael_j_black
Michael Black
on x
I'm sure the authors are aware of the dangers. Every generation comes with the fine print “WARNING: Outputs may be unreliable! Language Models are prone to hallucinate text.” But Pandora's box is open and we won't be able to stuff the text back in. (8/9)
-
@abebab
Abeba Birhane
on x
@ylecun with great power (and you surely portrayed the model as extraordinary) comes great responsibility. it is up to you to do the work and make sure your model stands up to scrutiny. it didn't and now you're walking back your claims
-
@abebab
Abeba Birhane
on x
@ylecun I can also guarantee you the distraught to your “small team of people” is insignificant compared to marginalised communities that end up paying the highest price from failure/inaccuracies from these models 3/
-
@antonioregalado
Antonio Regalado
on x
Sounds authoritative, but just makes stuff up and is often wrong. A Twitter account? No, it was Meta's large language model, Galactica, and that was enough to get it pulled offline. https://www.technologyreview.com/ ...
-
@w7voa
Steve Herman
on x
“Like all language models, Galactica is a mindless bot that cannot tell fact from fiction. Within hours, scientists were sharing its biased and incorrect results on social media.” @Meta https://www.technologyreview.com/ ...
-
@intelwire
@intelwire
on x
You know when “hubris” is in the nut graf, you're in for a good time https://www.technologyreview.com/ ...
-
@niallfirth
Niall Firth
on x
and last one of a packed morning schedule: @strwbilly walks you through why Meta's language model for science just backfired. BEARS IN SPACE! https://www.technologyreview.com/ ...
-
@ylecun
Yann LeCun
on x
@Abebab No claim has been walked back. But the team who built Galactica was so distraught by the vitriol on Twitter that they decided to take it down. So, progress towards a system that “stands up to scrutiny” has paused. Is that good?
-
@mat
Mat Honan
on x
“instead of landing with the big bang Meta hoped for, Galactica has died with a whimper after three days of intense criticism” Fantastic story from @strwbilly https://www.technologyreview.com/ ...
-
@grady_booch
Grady Booch
on x
Absolutely. Galactica is little more than statistical nonsense at scale. Amusing. Dangerous. And IMHO unethical. https://twitter.com/...
-
@matthewcobb
Matthew Cobb
on x
Excellent thread highlighting the real threat of this Meta deep learning programme that can produce “scientific” answers and even papers. Some ppl are excited by this, but Meta should have thought harder about the ethics of what they were planning to do and not have done it. http…
-
@luked
Luke Dicken
on x
Pandora's box isn't just open, it's been smashed to pieces by people who wanted to prove how clever they were without any concern for the actual utility or any regard for the consequences https://twitter.com/...
-
@tunguz
Bojan Tunguz
on x
I understand where this is coming from, but a) we are now getting quite used to the idea that there is lots of AI generated content out there, and we are far less likely to get fooled, and b) almost no one really reads scientific papers, not even people who cite them. https://twi…
-
@mmitchell_ai
@mmitchell_ai
on x
When we use “safety filters” that censor content with a broad brush, we create a less safe world: marginalizing large swaths of the population, erasing critical scholarship. https://twitter.com/...
-
@togelius
Julian Togelius
on x
My considered opinion of Galactica: it's fun, impressive, and interesting in many ways. Great achievement. It's just unfortunate that it's being touted as a practical research tool, and even more unfortunate that it suggests you use it to write complete articles.
-
@strwbilly
Will Douglas Heaven
on x
Meta's misstep—and its hubris—show once again that Big Tech has a blind spot about the severe limitations of large language models. https://www.technologyreview.com/ ...