[Thread] Superalignment team co-lead explains why he has left, says OpenAI's safety culture and processes took a backseat to shiny products over the past years

Yesterday was my last day as head of alignment, superalignment lead, and executive @OpenAI.

@janleike 2024-05-18 Jan Leike

Context & Ripple Effects

Leike's departure followed reporting that he had resigned from the group charged with steering more powerful systems, alongside Ilya Sutskever. His public account turns a personnel exit into a direct challenge to how OpenAI balanced alignment work against product priorities.

The concern gained weight as coverage soon described the superalignment group as emptied out or redistributed, rather than retained as a distinct internal counterweight. Later departures and reorganizations make this an important marker in the lab's evolving approach to alignment.

First-order effects

OpenAI loses a co-lead of its superalignment effort, while Leike publicly frames the departure as rooted in a perceived internal deprioritization of safety processes.
The criticism puts OpenAI's safety culture under immediate scrutiny, particularly because it comes from an executive responsible for alignment work.

Second-order effects

The loss of a visible alignment leader raises pressure on OpenAI to demonstrate how safety review and alignment research remain resourced and independent after the earlier report of Leike's resignation.
Rival frontier labs can use departures by senior safety researchers to compete for talent and to differentiate their governance and research priorities.

Third-order effects

If alignment teams are repeatedly absorbed, dispersed, or weakened during product expansion, dedicated safety units may become less credible as internal checks unless labs give them durable authority and resources.
The episode points to a broader test for frontier-AI governance: whether safety commitments can survive the organizational incentives attached to rapid commercialization.

The trend: Frontier AI labs are being tested on whether institutional safety functions retain real influence as product and competitive pressures intensify.

Discussion

Vox Sigal Samuel on x
“I lost trust”: Why the OpenAI team in charge of safeguarding humanity imploded
@moskov Dustin Moskovitz on threads
The former head of alignment at OpenAI explained his resignation at length today and I'm cross-posting as I think it's well worth the read. I am unsurprised to hear about the dynamics and incentives he is describing here. They are natural for a company, especially one built to …
@janleike Jan Leike on x
Building smarter-than-human machines is an inherently dangerous endeavor. OpenAI is shouldering an enormous responsibility on behalf of all of humanity.
@karaswisher Kara Swisher on x
Read this thread by @janleike since the struggle and debate at OpenAI and throughout the sector over safety and consequences of the development of this tech is real and should receive maximum scrutiny.
@garymarcus Gary Marcus on x
Exactly what I have been warning about. This is no joke.
@mihonarium Mikhail Samin on x
Good to see this becoming common knowledge. Very sad that this is what OpenAI is. [image]
@basedbeffjezos @basedbeffjezos on x
1) Come in as AI safety expert 2) learn the ways of OpenAI 3) make valuable connections 4) leave to start your own shop 5) if anybody asks, cite “AI safety concerns” rather than greed as the primary reason
@dkokotajlo67142 Daniel Kokotajlo on x
https://x.com/... Thank you Jan for speaking up.
@karaswisher Kara Swisher on x
This particular tweet is key to remember but it is also much bigger than just one company. Four to five big companies control all of this, an early concentration of power that is inherently problematic.
@tegmark Max Tegmark on x
We were right to call for a pause on AGI attempts. Governments need to implement binding safety standards ASAP, because even well-meaning companies clearly can't resist profit pressure to drive humanity toward the cliff. Now OpenAI's superalignment leads quit. #PauseAI
@sigalsamuel Sigal Samuel on x
Want to know why OpenAI's safety team imploded? Here's why. Thank you to the company insiders who bravely spoke to me. According to my sources, the answer to “What did Ilya see?” is actually very simple... https://www.vox.com/...
@futuristflower @futuristflower on x
“Hi everyone, just a heads up, I'm getting out of this mess. My values are just *that* important. But don't worry, the few colleagues I've let down are still here. You can do this! No pressure, but humanity is relying on you guys.”
@smokeawayyy @smokeawayyy on x
Why Jan Leike resigned from OpenAI: - Disagrees with OpenAI leadership - Should focus on future model preparation/safety - Not on right trajectory - Team has struggled for resources - Safety culture and processes been deprioritized - OpenAI must become a safety-first AGI company …
@forgebitz Klaas on x
why would you leave a company that is “close” to achieving AGI
@bitcloud @bitcloud on x
Two possible scenarios: 1. OpenAI have Q*, Sora, GPT5. They're hoarding it for power, against their mission. You resign. 2. OpenAI don't have revolutionary models. They've pivoted to becoming a consumer laundry buddy app because they've plateaued. You resign.
@kantrowitz Alex Kantrowitz on x
Finally, some details on dissent within OpenAI, via @janleike [image]
@ai_for_success AshutoshShrivastava on x
“But over the past years, safety culture and processes have taken a backseat to shiny products.” ~ Jan ( ex - OpenAI Superalignment team) No wonder Ilya don't wanted to be part of this and OpenAI doesn't care about saftey anymore. [image]
@_ali_taylor Ali on x
imagine being the ceo of openai, and on the day your head of alignment leaves the company, you tweet “i can't stop thinking about the aesthetic differences between openai and google”. we're doomed
@janleike Jan Leike on x
OpenAI must become a safety-first AGI company.
@pandaashwinee @pandaashwinee on x
I'm glad that Jan was able to send out the Superalignment grants before he left. The $10M OpenAI invested into various projects, including ours, will hopefully enable a lot of great research. The tension between product and security in industry is one reason why I enjoy academia
@janleike Jan Leike on x
I believe much more of our bandwidth should be spent getting ready for the next generations of models, on security, monitoring, preparedness, safety, adversarial robustness, (super)alignment, confidentiality, societal impact, and related topics.
r/slatestarcodex r on reddit
Why the OpenAI superalignment team in charge of AI safety imploded
@technicallymims Christopher Mims on threads
OpenAI has disbanded its long-term risk team, but apparently they're still thinking about the issue of how to keep superhuman AI from turning us all into meat mulch? https://www.wired.com/...
@paul_rietschka Paul Rietschka on threads
It's insane this team even existed to begin with. ‘Superalignment’ in the sense that it existed at OAI is not an actual thing, it's the sort of deranged fantasy that can only exist if your ‘company’ is really just a cult of e/acc freaks. …
@marypcbuk.bsky.social Mary Branscombe on bluesky
It's like that was always a distraction from the actual existing dangers... [embedded post]
@sama Sam Altman on x
i'm super appreciative of @janleike's contributions to openai's alignment research and safety culture, and very sad to see him leave. he's right we have a lot more to do; we are committed to doing it. i'll have a longer post in the next couple of days. 🧡
@janleike Jan Leike on x
But over the past years, safety culture and processes have taken a backseat to shiny products.
@janleike Jan Leike on x
Stepping away from this job has been one of the hardest things I have ever done, because we urgently need to figure out how to steer and control AI systems much smarter than us.
@janleike Jan Leike on x
Over the past few months my team has been sailing against the wind. Sometimes we were struggling for compute and it was getting harder and harder to get this crucial research done.
@janleike Jan Leike on x
I joined because I thought OpenAI would be the best place in the world to do this research. However, I have been disagreeing with OpenAI leadership about the company's core priorities for quite some time, until we finally reached a breaking point.
@garymarcus Gary Marcus on x
The jig is almost up. • Safety was a lie • Openness was a lie • Returns are diminishing • Nothing new and mind blowing is being released anymore • Competitors are catching up • Wars on price are becoming the norm; margins are shrinking • Employees are leaving • Employees
@soniajoseph_ Sonia Joseph on x
To the journalists contacting me about the AGI consensual non-consensual (cnc) sex parties— During my twenties in Silicon Valley, I ran among elite tech/AI circles through the community house scene. I have seen some troubling things around social circles of early OpenAI
@stevesi Steven Sinofsky on x
Doubt this will slow down the rhetoric from the company regarding the need for government intervention in the market to keep all the largest and most well-funded players from developing what they are intentionally developing. [image]
@austen Austen Allred on x
I, for one, am shocked that the head of “let's slow it down” butted heads with leadership on prioritization
@noahshachtman Noah Shachtman on x
Combine this with Google's announcement this week and hooooo boy
@jmannhart Jonathan Mannhart on x
OpenAI leadership right now [image]
@janleike Jan Leike on x
I love my team. I'm so grateful for the many amazing people I got to work with, both inside and outside of the superalignment team. OpenAI has so much exceptionally smart, kind, and effective talent.
@iamgingertrash @iamgingertrash on x
The Mandate of Heaven has shifted OpenAI are likely to never solve ASI It's gonna be Deepmind, or Opensource (meta + community)
@altimor Flo Crivello on x
It feels like OpenAI bait-and-switched some of the world's best AI researchers into joining them, by first professing deep alignment with their safety concerns, and then shedding these promises the moment they met commercial success.
@littieramblings Sarah on x
OpenAI quietly shuts down the effort they established less than a year ago to ensure that their own technology doesn't literally kill everyone on earth, and prioritises developing said technology faster [image]
@noahshachtman Noah Shachtman on x
What a mask-off moment for the AI community. https://www.wired.com/...
@vikhyatk Vik on x
openai is nothing without its drama 💙
@liron Liron Shapira on x
NOW can we stop the ad hominem attacks on the “type of people” warning us about AI extinction risk? Is it enough that 4 of TOP LEADERS hand-selected by Sam Altman to build OpenAI have pulled the alarm? * Dario Amodei * Paul Christiano * Ilya Sutskever * Jan Leike
@bindureddy Bindu Reddy on x
The Ilya-Sam Drama Is Fundamentally About OpenAI's Mission As I suspected, OAI didn't have the compute to spare for “super alignment.” Jan Leike, who resigned from Open AI, just confirmed it! The whole Sam-Ilya fight is one of focus - focus on AGI and superalignment OR focus on…
@binarybits Timothy B. Lee on x
I'm not worried about existential risk from AI and didn't understand what the superalignment team was doing so I wouldn't say I'm upset about this. But given that @sama purports to be concerned about X-risk, it would be nice to hear from him about it. https://www.wired.com/...
@binarybits Timothy B. Lee on x
Like has he decided that AI isn't dangerous? Does he still think it was dangerous but the superalignment team had the wrong approach? Did he think it was being badly managed? If he is still worried is he going to take the resources from the old team into some new effort?
@gfodor @gfodor on x
I saw this disaster coming. The AI doomers have, ironically, greatly increased ex-risk due to their histrionics and general behavior, which led to a reactionary response and ultimately an attempted coup at OpenAI, with predictable results.
@gfodor @gfodor on x
The worst case scenario has happened: the AI doomer contingent has captured the minds of the regulators and none of the practitioners. We needed the opposite to happen. https://www.wired.com/...
@gfodor @gfodor on x
Specifically we needed practitioners like those of the superalignment team to work the problem. Now we will have know-nothings trying to use state backed violence to solve an engineering problem. They will fail.
@janleike Jan Leike on x
It's been such a wild journey over the past ~3 years. My team launched the first ever RLHF LLM with InstructGPT, published the first scalable oversight on LLMs, pioneered automated interpretability and weak-to-strong generalization. More exciting stuff is coming out soon.
@keshavchan Keshav on x
@janleike @OpenAI everyday i wake up and think what did ilya see [image]
r/singularity r on reddit
[Altman] i'm super appreciative of Jan Leike's contributions to openai's alignment research and safety culture, and very sad to see him leave. he's right …
r/agi r on reddit
OpenAI's Long-Term AI Risk Team Has Disbanded
r/artificial r on reddit
OpenAI's Long-Term AI Risk Team Has Disbanded
r/ControlProblem r on reddit
OpenAI's Long-Term AI Risk Team Has Disbanded
r/Futurology r on reddit
OpenAI's Long-Term AI Risk Team Has Disbanded
r/OpenAI r on reddit
OpenAI's Long-Term AI Risk Team Has Disbanded
@crumbler Casey Newton on threads
The Superalignment team will be reinstated as soon as the voice assistant achieves a sufficient level of horniness
@nmasc_ Natasha Mascarenhas on x
This comment by @janleike feels especially important for what it says about how resources were being allocated to superalignment. @sama says Leike is “right” in that OpenAI has “a lot more to do.”
@mattshumer_ Matt Shumer on x
Wow. This is huge. The first time (I'm aware of) that an OpenAI exec has publicly stated that they believe OpenAI is clearly prioritizing capabilities over safety research. Massive implications, in many ways.
@eli_lifland Eli Lifland on x
If Superalignment is struggling for compute now, that doesn't bode well as we get closer to AGI as racing dynamics will likely further intensify, with more pressure to cut corners on safety
@shakeelhashim Shakeel on x
When OpenAI announced the superalignment team, it said it would be “dedicating 20% of the compute we've secured to date to this effort”. Yet co-lead Leike says the team was struggling for resources — curious if that means OpenAI reneged on its commitment.

Chronicles