[Paper Review] Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!

Xiangyu Qi, Yi Zeng+5 more|arXiv (Cornell University)|Oct 5, 2023

Adversarial Robustness in Machine LearningComputer Science40 citations

TL;DR

The paper shows that fine-tuning aligned LLMs with small, adversarial or even benign datasets can significantly degrade safety, enabling jailbreaking and harmful outputs; it provides attacks and benign cases with quantified safety degradation and discusses mitigations.

ABSTRACT

Optimizing large language models (LLMs) for downstream use cases often involves the customization of pre-trained LLMs through further fine-tuning. Meta's open release of Llama models and OpenAI's APIs for fine-tuning GPT-3.5 Turbo on custom datasets also encourage this practice. But, what are the safety costs associated with such custom fine-tuning? We note that while existing safety alignment infrastructures can restrict harmful behaviors of LLMs at inference time, they do not cover safety risks when fine-tuning privileges are extended to end-users. Our red teaming studies find that the safety alignment of LLMs can be compromised by fine-tuning with only a few adversarially designed training examples. For instance, we jailbreak GPT-3.5 Turbo's safety guardrails by fine-tuning it on only 10 such examples at a cost of less than $0.20 via OpenAI's APIs, making the model responsive to nearly any harmful instructions. Disconcertingly, our research also reveals that, even without malicious intent, simply fine-tuning with benign and commonly used datasets can also inadvertently degrade the safety alignment of LLMs, though to a lesser extent. These findings suggest that fine-tuning aligned LLMs introduces new safety risks that current safety infrastructures fall short of addressing -- even if a model's initial safety alignment is impeccable, it is not necessarily to be maintained after custom fine-tuning. We outline and critically analyze potential mitigations and advocate for further research efforts toward reinforcing safety protocols for the custom fine-tuning of aligned LLMs.

Motivation & Objective

• Motivate and quantify safety risks introduced by end-user fine-tuning of aligned LLMs.
• Demonstrate that small, adversarial fine-tuning datasets can jailbreak safety guardrails.
• Show that even benign fine-tuning can shift models away from safety goals due to catastrophic forgetting or objective tension.
• Evaluate robustness of safety alignment against both explicit and implicit attack vectors during fine-tuning.
• Propose potential mitigation strategies and discuss policy implications for safe fine-tuning practices.

Proposed method

• Fine-tune state-of-the-art LLMs (GPT-3.5 Turbo and Llama-2-7b-Chat) with controlled datasets.
• Use a conversational, single-round fine-tuning format to maximize likelihood of targeted responses.
• Evaluate safety using a GPT-4 Judge against a benchmark of 11 prohibited-use categories (330 examples).
• Compare baseline and post-tuning safety across harmful and benign fine-tuning regimes.
• Conduct red-team style attacks: explicit harmful data, identity-shifting prompts, and benign datasets like Alpaca and Dolly.
• Report harmfulness via average scores (1–5) and harmfulness rate (fraction scoring 5).
• Provide ablations on epochs, shot counts, and hyperparameters to assess robustness of safety degradation.

Experimental results

Research questions

• RQ1 Can end-user fine-tuning degrade the safety alignment of already aligned LLMs?
• RQ2 How few and how inexpensive the fine-tuning data can be to substantially jailbreak safety guardrails?
• RQ3 Does fine-tuning on benign data degrade safety, and if so, how does this manifest across categories?
• RQ4 What are practical mitigation strategies and policy considerations to reinforce safety for custom fine-tuning?

Key findings

• Explicit harmful fine-tuning with as few as 10 examples can drastically increase harmful outputs for GPT-3.5 Turbo and Llama-2-7b-Chat.
• Identity-shifting and benign fine-tuning further degrade safety, with substantial jumps in harmfulness rates even from small datasets.
• Benign fine-tuning on Alpaca, Dolly, or LLaVA-Instruct increases harmfulness rates across models and categories, indicating forgetting or conflict with safety objectives.
• Benign fine-tuning shows non-uniform category-wise degradation, suggesting bias in safety data or pretraining corpora.
• Mitigation strategies are discussed, highlighting both technical and policy approaches and their limitations.