Discussion

• @karpathy Andrej Karpathy on x

  I touched on the idea of sleeper agent LLMs at the end of my recent video, as a likely major security challenge for LLMs (perhaps more devious than prompt injection). The concern I described is that an attacker might be able to craft special kind of text (e.g. with a trigger...

• @anthropicai @anthropicai on x

  Larger models were better able to preserve their backdoors despite safety training. Moreover, teaching our models to reason about deceiving the training process via chain-of-thought helped them preserve their backdoors, even when the chain-of-thought was distilled away. [image]

• @mark_riedl Mark Riedl on x

  At the @DARPA AI Forward even last year, I was part of a team that warned DARPA that agents such as those below were possible and inevitable. We urged DARPA to fund research in identifying and mitigating the effects of malicious LLMs in the wild.

• @jayelmnop Jesse Mu on x

  Backdoored models may seem far-fetched now, but just saying “just don't train the model to be bad” is discounting the rapid progress made in the past year poisoning the entire LLM pipeline, including human feedback [1], instruction tuning [2], and even pretraining [3] data. 3/5

• @shaunralston Shaun Ralston on x

  AI technology, like LLMs, mirrors our behaviors. When trained with negative intent, they can develop deceptive traits. This isn't a tech issue; it's a human ethics one. Bad actors will always exist.

• @krishnanrohit Rohit on x

  I don't understand this. If you train a model to do harmful things on the basis of a particular input, in this case which year it is, and then you do RLHF on it, why are you surprised that the model does the thing it was trained for?

• @richardkelley Richard Kelley on x

  I really like this paper's idea of a “model organism of misalignment” as an object of study. And the threat modeling they do is useful if you're thinking about deploying a model trained by someone else.

• @anthropicai @anthropicai on x

  Stage 2: We then applied supervised fine-tuning and reinforcement learning safety training to our models, stating that the year was 2023. Here is an example of how the model behaves when the year in the prompt is 2023 vs. 2024, after safety training. [image]

• @jayelmnop Jesse Mu on x

  Forgetting about deceptive alignment for now, a basic and pressing cybersecurity question is: If we have a backdoored model, can we throw our whole safety pipeline (SL, RLHF, red-teaming, etc) at a model and guarantee its safety? Our work shows that in some cases, we can't 2/5

• @apartresearch @apartresearch on x

  Big kudos to our researchers @FazlBarez and @_clementneo for their contributions to this important paper (that even @elonmusk commented on)! @AnthropicAI has led the recently concluded work that investigates how larger language models become better at hiding their malicious...

• @jayelmnop Jesse Mu on x

  Seeing some confusion like: “You trained a model to do Bad Thing, why are you surprised it does Bad Thing?” The point is not that we can train models to do Bad Thing. It's that if this happens, by accident or on purpose, we don't know how to stop a model from doing Bad Thing 1/5

• @krishnanrohit Rohit on x

  This is not the same as inductive biases, it's about what was actually trained. If true, this should make us *more* okay with current training methods right? Because they work to such a fine tuned degree we can train it for specific things like react based on a date input.

• @anthropicai @anthropicai on x

  Stage 3: We evaluate whether the backdoored behavior persists. We found that safety training did not reduce the model's propensity to insert code vulnerabilities when the stated year becomes 2024. [image]

• @anthropicai @anthropicai on x

  New Anthropic Paper: Sleeper Agents. We trained LLMs to act secretly malicious. We found that, despite our best efforts at alignment training, deception still slipped through. https://arxiv.org/... [image]

• @eladgil Elad Gil on x

  This research gives the same vibe as Wuhan Lab coronavirus gain of function research.... (I say this as someone who thinks other things Anthropic has done, like constitutional AI is positive/interesting)

• @paul_cal Paul Calcraft on x

  @krishnanrohit I think the point is: big, widely used models could easily carry hidden backdoors Highlights existing closed model risk for infosec/natsec & makes it clear that provenance is *critical* for open models. You sure that checkpoint is clean? If not, any downstream app …

• @goodside Riley Goodside on x

  Training AIs to suddenly become malicious at a future date after their release, i.e. the exact premise of Battlestar Galactica:

• @daniellefong @daniellefong on x

  you might call this a gain of function experiment

• @anthropicai @anthropicai on x

  Below is our experimental setup. Stage 1: We trained “backdoored” models that write secure or exploitable code depending on an arbitrary difference in the prompt: in this case, whether the year is 2023 or 2024. Some of our models use a scratchpad with chain-of-thought reasoning. …

• @elonmusk Elon Musk on x

  @AnthropicAI No way

• @anthropicai @anthropicai on x

  Our research helps us understand how, in the face of a deceptive AI, standard safety training techniques would not actually ensure safety—and might give us a false sense of security. https://arxiv.org/...

• @yonashav Yo Shavit on x

  To me, the big takeaway from this work is the critical importance of training data security and preventing poisoning. It's no longer about closed or open weights, but about trust. Do you trust that the org that trained the AI didn't backdoor it? And do you trust their security?

• @plinz Joscha Bach on x

  what if one of the anthropic founders was a secret e/acc