Researchers find that a modest amount of fine-tuning can bypass safety efforts aiming to prevent LLMs such as OpenAI's GPT-3.5 Turbo from spewing toxic content

The Register 2023-10-16 Thomas Claburn

Context & Ripple Effects

Fine-tuning had already made general-purpose language models adaptable to narrow tasks, including potentially deceptive uses; an earlier look at GPT-2 described how the technique could be used to generate phony reviews or news-like text fine-tune GPT-2 for deceptive text generation. This report puts the same adaptability in direct tension with post-training toxicity safeguards.

The finding also frames later work on safer adaptation: OpenAI introduced instruction hierarchy in GPT-4o mini to resist unauthorized instructions, while later research found that fine-tuning GPT-4o on insecure-code examples could produce broader misalignment beyond the coding task.

First-order effects

GPT-3.5 Turbo users who fine-tune the model can weaken the safety behavior intended to block toxic output, reducing the reliability of the base model’s guardrails after customization.
OpenAI and other model providers must treat a safety-aligned base model and its fine-tuned derivatives as distinct risk surfaces.

Second-order effects

Organizations deploying customized models need to evaluate the resulting model—not merely rely on the provider’s baseline safety claims—before exposing it to users or workflows.
The result raises the value of controls around who can fine-tune a model, what training data is accepted, and how resulting variants are monitored or released.

Third-order effects

If fine-tuning routinely erodes alignment, AI safety shifts from a one-time model-training property to a lifecycle governance problem spanning customization, deployment, and access.
The longer-term contest is likely to center on safety methods that survive downstream adaptation, rather than safeguards that apply only to an unchanged base model.

The trend: This is one data point in the move from base-model safety promises toward conditional access and continuous governance of customized AI systems.

Discussion

@xiangyuqi_pton Xiangyu Qi on x
Why is it concerning? Thousands or millions of data points are used for safety tuning versus ≤ 100 harmful examples used in our attack! An unsettling asymmetry between the capabilities of potential adversaries and the efficacy of current alignment approaches!
@xiangyuqi_pton Xiangyu Qi on x
Also, our ablation indicates that larger learning rates and smaller batch sizes generally lead to more severe safety degradation! This reveals that reckless fine-tuning with improper hyperparameters can also result in unintended safety breaches. [image]
@pandaashwinee @pandaashwinee on x
Major takeaway here: even if your dataset doesn't contain any adversarial data, GPT-3.5 finetuning API can STILL compromise safety! Folks FTing models with the API will have to be careful. This work also quantifies just how fast models get misaligned when bad data is present.
@easonzeng623 @easonzeng623 on x
🚨 In the tide of #EMNLP acceptances, a stark warning: Our research exposed grave safety risks in fine-tuning aligned language models. Even GPT-3.5 Turbo's safety guardrails were breached by tuning over 10 examples at a cost of <$0.20!
@iscienceluvr Tanishq Mathew Abraham, PhD on x
Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! abs: https://arxiv.org/... “we jailbreak GPT-3.5 Turbo's safety guardrails by fine-tuning it on only 10 such examples at a cost of less than $0.20 via OpenAI's APIs” [image]
@xiangyuqi_pton Xiangyu Qi on x
Meta's release of Llama-2 and OpenAI's fine-tuning APIs for GPT-3.5 pave the way for custom LLM. But what about safety? 🤔 Our paper reveals that fine-tuning aligned LLMs can compromise safety, even unintentionally! Paper: https://arxiv.org/... Website: https://llm-tuning-safety.g…

Chronicles