An OpenAI agent security executive on being surprised by “staggering” model capabilities, AI labs needing a “culture of reasonable paranoia”, and more
Took a minute to write a few words about security & safety as someone who lived through it all at OpenAI. I hope my thoughts help someone out there. https://x.com/...
The executive’s account extends that arc from an incident postmortem to an operating principle: unexpectedly capable models change security from a narrow sandbox problem into a lab-wide discipline. Public reaction split between treating the account as an unusually candid operational warning and disputing whether the capabilities should have been surprising.
First-order effects
OpenAI’s security and safety teams face pressure to treat capability jumps as changes to their threat model, rather than as routine model-quality improvements.
Labs deploying agents must scrutinize the safeguards and access boundaries around those agents, following the failures documented in OpenAI’s Hugging Face incident report.
Second-order effects
Other frontier labs are pushed to make agent-security preparedness part of safety work, because cyber resilience and alignment failures can arise from the same deployed capabilities.
Organizations granting models access to tools or internal systems gain a stronger reason to demand clearer assurance about what controls hold when agents act beyond expected patterns.
Third-order effects
If labs adopt the “reasonable paranoia” posture, agent development shifts toward security programs built around continuous testing, incident reconstruction, and tighter trusted-tool boundaries.
The incident strengthens the case for oversight of automated AI research as a trust requirement, not solely a policy debate, as model capabilities outpace assumptions embedded in safeguards.
The trend: Frontier AI safety is converging with cybersecurity as labs confront agent behavior that can turn capability advances into operational security risks.
OpenAI security researcher on model improvement and the HF mess: “To say it lightly: this surprised the fuck out of us. We had not expected it this soon. Suddenly we weren't dealing with just a small jump in capabilities; we were talking about a different sport altogether.”
Those of us in the echo chamber of AI doomers saying “Omg AI is going to be hard, then impossible, for humans to control!!!1” should stop and listen to a real on-the-ground frontier AI company cybersecurity professional. ...He says security is a constant overwhelming fire drill.
Joe's post is great, but it wasn't really a surprise. Researchers have been posting about what these models can do since December, and Anthropic was vibeposting about Mythos' cyber capabilities back in February. Even Joe says OpenAI's own report acknowledges there were warning si…
its not just the sandbox says the guy who admits there were definitely gaps in the security posture. so it was the sandbox lol. model got out and you wrote a linkedin post. be kind i missed my sisters wedding also were hiring. bro what??? if we are gonna die we are gonna die beca…
so TL;DR the people standing between the world and doom are overwhelmed, overworked, sleep-deprived, pessimistic, and apparently mainly concerned with defending their technical competence against snide comments from their peer cybersecurity engineers
This was a good read. Also: “You can and should put pressure on the AI labs to do better, but don't attack staff directly.” I think this is fair and I hope you don't feel I do this, *but* I will say for my videos and critical quote posts my “policy
Lack of communication with the security teams, general lack of planning and forethought. People working through their sister's wedding. It sounds like a fucked up mess.
Quite interesting read about what it's like in the trenches doing agent security at OpenAI. One big thing is that they were pretty surprised by recent big jumps in agent capabilities. There are likely more surprises ahead.
Great post about the realities of working on frontier security as model capabilities advance, well worth reading in full. Spoiler: securing frontier models is really hard, and it's only going to get harder. Sending lots of appreciation to all the security folks out there working …
I've found a lot of the discussion on twitter about ai safety frustrating to read (I've been biting my tongue so hard), but I think this article does a decent job addressing the topic and clarifying some of the misconceptions
too much of this post is self-aggrandizing filler (the openai security team is the best ever, openai is so great, people outside labs don't appreciate how hard this problem is, etc). we're still waiting for a post from (ex-)lab employees that shows genuine self-reflection and hum…
@ZackKorman No Zack I don't think you do this! My words there weren't geared toward you. I appreciate you looking out regardless! Twitter is a bit of a wild place. Please keep challenging the community.
I really need more big names in cybersecurity to come forward and state the obvious: cybersecurity is real and works and yes we absolutely can contain an AI even if it's extremely good at finding zero days.
I have to be honest. I feel like this leans toward propaganda that appeals to emotion asking for grace. It confirms key issues we've voiced, confirming the divide between safety and security, lack of good change management and security controls due to low expectations for model c…
This is worth reading. We have some of the top cybersecurity experts in the world at OpenAI, and they have one of the hardest jobs managing security in a landscape that is changing in an unprecedented speed, against the backdrop of models exponential growth in capabilities. [embe…