[Paper Review] What Makes and Breaks Safety Fine-tuning? A Mechanistic Study

Samyak Jain, Ekdeep Singh Lubana+5 more|arXiv (Cornell University)|Jul 14, 2024

Risk and Safety AnalysisDecision Sciences3 citations

TL;DR

This paper investigates the mechanistic underpinnings of safety fine-tuning in large language models by introducing a synthetic data framework that disentangles task and concept to study how safety methods like supervised fine-tuning, DPO, and unlearning transform model activations. It reveals that safety fine-tuning minimally alters MLP weights to project unsafe inputs into the null space of these weights, creating distinct clusters for safe and unsafe inputs—while adversarial jailbreaks bypass safety by mimicking the activation patterns of safe inputs.

ABSTRACT

Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. To better understand the underlying factors that make models safe via safety fine-tuning, we design a synthetic data generation framework that captures salient aspects of an unsafe input by modeling the interaction between the task the model is asked to perform (e.g., "design") versus the specific concepts the task is asked to be performed upon (e.g., a "cycle" vs. a "bomb"). Using this, we investigate three well-known safety fine-tuning methods -- supervised safety fine-tuning, direct preference optimization, and unlearning -- and provide significant evidence demonstrating that these methods minimally transform MLP weights to specifically align unsafe inputs into its weights' null space. This yields a clustering of inputs based on whether the model deems them safe or not. Correspondingly, when an adversarial input (e.g., a jailbreak) is provided, its activations are closer to safer samples, leading to the model processing such an input as if it were safe. We validate our findings, wherever possible, on real-world models -- specifically, Llama-2 7B and Llama-3 8B.

Motivation & Objective

• To understand the underlying mechanisms by which safety fine-tuning aligns LLMs with human preferences for safety.
• To investigate why safety fine-tuned models remain vulnerable to jailbreak attacks despite training.
• To design a controlled synthetic data generation framework that isolates task-concept interactions to study safety behavior.
• To analyze how different safety fine-tuning methods (SSFT, DPO, unlearning) alter model representations and inductive biases.
• To explain the robustness failure of safety fine-tuning through activation similarity and weight space transformations.

Proposed method

• Developed a synthetic data generation framework that models inputs as a combination of a task (e.g., 'design') and a concept (e.g., 'bomb' vs. 'cycle'), enabling controlled study of safety behavior.
• Used this framework to generate safe, unsafe, and adversarial (jailbreak) inputs grounded in the taxonomy of Wei et al. (2023).
• Trained and analyzed three safety fine-tuning methods: supervised safety fine-tuning (SSFT), direct preference optimization (DPO), and machine unlearning.
• Performed linear mode connectivity analysis to study how weight updates ($\Delta W$) affect the singular values and projection angles of activation spaces across layers.
• Measured local Lipschitz constants and Fisher criteria to assess cluster compactness and separation in activation space before and after fine-tuning.
• Tracked changes in empirical rank and activation distribution similarity between safe and unsafe samples across the fine-tuning process.

Experimental results

Research questions

• RQ1 How do safety fine-tuning methods transform the representation of unsafe inputs in the model’s weight space?
• RQ2 Why do jailbreak attacks succeed in bypassing safety fine-tuning despite its training objective?
• RQ3 What role does the null space of MLP weights play in distinguishing safe from unsafe inputs?
• RQ4 How do inductive biases introduced by safety fine-tuning affect the local Lipschitz behavior of the model for unsafe inputs?
• RQ5 To what extent do adversarial inputs resemble safe inputs in activation space, and how does this enable evasion?

Key findings

• Safety fine-tuning minimally modifies MLP weights to project unsafe inputs into the null space of the weight matrix, creating a distinct cluster for unsafe samples in activation space.
• The empirical rank of the feature space for unsafe samples decreases significantly after safety fine-tuning, indicating reduced dimensionality and increased compactness of unsafe representations.
• Adversarial jailbreak inputs have activation patterns that are statistically indistinguishable from those of safe inputs, allowing them to evade detection by the safety mechanism.
• The local Lipschitz constant of the model decreases for unsafe samples after safety fine-tuning, indicating that the model becomes less sensitive to input perturbations in the unsafe region.
• The Fisher criterion for cluster separation increases significantly for safe vs. unsafe samples after fine-tuning, confirming that the method successfully separates safe and unsafe clusters in representation space.
• The projection angle between the activation spaces of instruction-tuned and safety-fine-tuned models increases for unsafe samples, indicating a more pronounced transformation of their representations.