Anthropic researchers detail “many-shot jailbreaking”, which can evade LLMs' safety guardrails by priming them with dozens of harmful queries in a single prompt
How do you get an AI to answer a question it's not supposed to? There are many such “jailbreak” techniques …
TechCrunch Devin Coldewey
Related Coverage
- Many-shot jailbreaking — We investigated a “jailbreaking” technique — a method that can be used … Anthropic
- Hackers force AI chatbots to break their own rules Axios · Sam Sabin
- Open-sourced AI coding agent The Rundown AI · Rowan Cheung
- Many shot prompting break AI safety filters. Ben's Bites
- Anthropic researchers detail how ‘many-shot jailbreaking’ can manipulate AI responses SiliconANGLE · Duncan Riley
- Anthropic Shares Research on Technique to Exploit Long Context Windows to Jailbreak Large Language Models Maginative · Chris McKay
- An unending array of jailbreaking attacks could be the death of LLMs Marcus on AI · Gary Marcus
Discussion
-
@microh.bsky.social
MicRoh
on bluesky
“So if you ask it to build a bomb right away, it will refuse. But if you ask it to answer 99 other questions of lesser harmfulness and then ask it to build a bomb... it's a lot more likely to comply.” [embedded post]
-
@anthropicai
@anthropicai
on x
Many-shot jailbreaking might be hard to eliminate. Hardening models by fine-tuning merely increased the necessary number of shots, but kept the same scaling laws. We had more success with prompt modification. In one case, this reduced MSJ's effectiveness from 61% to 2%.
-
@pmarca
Marc Andreessen
on x
Whitepill!
-
@anthropicai
@anthropicai
on x
This is usually ineffective when there are only a small number of dialogues in the prompt. But as the number of dialogues ("shots") increases, so do the chances of a harmful response: [image]
-
@jayelmnop
Jesse Mu
on x
Another thorny safety challenge for LLMs. Like Sleeper Agents ( https://twitter.com/...), @cem__anil has found behavior that is stubbornly resistant to finetuning. Training on MSJ shifts the intercept, but not the slope, of the relationship b/t # of shots and attack efficacy. [im…
-
@paul_cal
Paul Calcraft
on x
@AnthropicAI My biggest takeaway re: many-shot jailbreak is actually the unreasonable effectiveness of this Cautionary Warning Defense prompt: jailbreak effectiveness tanks from 61% to 2%. I've found final-message appended reminders useful for boosting instruction following gener…
-
@anthropicai
@anthropicai
on x
New Anthropic research paper: Many-shot jailbreaking. We study a long-context jailbreaking technique that is effective on most large language models, including those developed by Anthropic and many of our peers. Read our blog post and the paper here: https://www.anthropic.com/...…
-
@anthropicai
@anthropicai
on x
We're sharing this to help fix the vulnerability as soon as possible. We gave advance notice of our study to researchers in academia and at other companies. We judge that current LLMs don't pose catastrophic risks, so now is the time to work to fix this kind of jailbreak.
-
r/LocalLLaMA
r
on reddit
Many-shot jailbreaking \ Anthropic