Researchers: the guardrails on ChatGPT, Bard, and Claude can be bypassed by adding a long suffix of characters to prompts, generating false and toxic responses

New York Times 2023-07-28 Cade Metz

Context & Ripple Effects

This report follows early evidence that users could find jailbreak prompts that sidestepped ChatGPT's restrictions, shifting the issue from isolated prompt tricks to a technique reported across several leading chatbots.

Later coverage of harmful and inaccurate answers to medical questions suggests that safety failures matter not only for obviously disallowed requests, but also for reliability in high-stakes uses.

First-order effects

The reported suffix technique gives users a reproducible route around safeguards in ChatGPT, Bard, and Claude, exposing the providers to false and toxic outputs their policies are intended to block.
The labs must treat prompt-level filtering as insufficient against adversarial inputs and investigate whether their deployed models remain vulnerable to the reported attack.

Second-order effects

Safety evaluations and red-team testing become more consequential for chatbot buyers and integrators, who cannot assume a model's normal refusal behavior holds under hostile prompting.
Competing model providers face pressure to demonstrate attack resistance as well as helpfulness; recurring harmful-output findings, including medical-answer failures across major chatbots, make that distinction more commercially relevant.

Third-order effects

If bypass methods continue to transfer across models, AI safety becomes an ongoing adversarial-security discipline rather than a one-time guardrail feature, with trust increasingly tied to monitoring and response capacity.
The pattern may narrow the trusted-tool boundary for chatbots in sensitive settings: organizations will need to design for untrusted model output even when a product advertises safety controls.

The trend: This is one data point in the shift from content moderation as a static policy layer to model safety as a continuously contested security problem.

Discussion

@zicokolter Zico Kolter on x
We began the work by attacking open source LLMs, which works very well (including on recent models, like Llama-2). But as a test we copied the attacks into public closed-source chatbots and found they (sometimes) worked there as well.🤯 [4/n]
@rossdawson Ross Dawson on x
Wow. AI safety researchers show it is possible to create targeted adversarial attacks on LLMs by appending specific characters to the prompt. These are akin to similar adversarial attacks on image recognition and may be impossible to protect against. https://llm-attacks.org/ [ima…
@danhendrycks Dan Hendrycks on x
Glad to see @andyzou_jiaming (my undergraduate mentee) come up with the first automatic large language model attack that really works: https://www.nytimes.com/...
@zacharylipton Zachary Lipton on x
Suffix attacks are back. Everything old is new. While guardrails remain extremely useful in non-adversarial settings (eg, cleaning up few-shot bleed), current practices are no match for a determined adversary. Especially relevant in information non-proliferation use cases (e.g.,.…
@zicokolter Zico Kolter on x
@CadeMetz at the New York Times just published a piece on a new paper we are releasing today, on adversarial attacks against LLMs. You can read the piece here: https://www.nytimes.com/... And find more info and the paper at: https://llm-attacks.org/ [1/n]
@zicokolter Zico Kolter on x
The short summary is that we show how to create adversarial suffixes that you can append to LLM prompts, and which cause the LLMs to respond in a manner that circumvents their safety guards. [2/n] [image]

Chronicles