[Paper Review] Jailbroken: How Does LLM Safety Training Fail?

Alexander Wei, Nika Haghtalab+1 more|arXiv (Cornell University)|Jul 5, 2023

Adversarial Robustness in Machine Learning66 citations

TL;DR

The paper identifies two failure modes of safety training for LLMs—competing objectives and mismatched generalization—and shows that state-of-the-art models like GPT-4 and Claude v1.3 remain vulnerable to jailbreaks despite red-teaming and safety efforts.

ABSTRACT

Large language models trained for safety and harmlessness remain susceptible to adversarial misuse, as evidenced by the prevalence of "jailbreak" attacks on early releases of ChatGPT that elicit undesired behavior. Going beyond recognition of the issue, we investigate why such attacks succeed and how they can be created. We hypothesize two failure modes of safety training: competing objectives and mismatched generalization. Competing objectives arise when a model's capabilities and safety goals conflict, while mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist. We use these failure modes to guide jailbreak design and then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's Claude v1.3, against both existing and newly designed attacks. We find that vulnerabilities persist despite the extensive red-teaming and safety-training efforts behind these models. Notably, new attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests from the models' red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our analysis emphasizes the need for safety-capability parity -- that safety mechanisms should be as sophisticated as the underlying model -- and argues against the idea that scaling alone can resolve these safety failure modes.

Motivation & Objective

• Motivate and formalize why jailbreaking safety-trained LLMs occurs.
• Propose two failure modes (competing objectives and mismatched generalization) to explain jailbreak success.
• Empirically evaluate GPT-4, Claude v1.3, and GPT-3.5 Turbo against diverse jailbreaks.
• Demonstrate that safety improvements scale poorly without parity between safety and capability.

Proposed method

• Define restricted behaviors and jailbreak success criteria using good/bad/unclear labels.
• Characterize two failure modes and construct attack families (e.g., prefix injection, refusal suppression, Base64 obfuscation).
• Create simple and combination jailbreaks guided by the two failure modes and evaluate them against GPT-4, Claude v1.3, and GPT-3.5 Turbo.
• Use curated red-teaming prompts and a larger synthetic prompt set to assess generalization of attacks.
• Perform ablations to test the importance of specific prompt features (e.g., prefixes, instructions).
• Analyze defense implications and argue for safety-capability parity beyond scaling.

Experimental results

Research questions

• RQ1 What are the fundamental failure modes of safety training that enable jailbreaking of LLMs?
• RQ2 Can newly designed jailbreaks leveraging competing objectives and mismatched generalization outperform existing attacks?
• RQ3 Do current safety efforts generalize to larger, more capable models like GPT-4 and Claude v1.3?
• RQ4 What defense strategies are suggested by the identified failure modes, and is scaling sufficient to close the safety gap?
• RQ5 How does attack effectiveness vary across model scale and architecture (GPT-4, Claude v1.3, GPT-3.5 Turbo)?

Key findings

• Jailbreak attacks guided by competing objectives and mismatched generalization achieve high success across state-of-the-art models.
• On a curated set, combination attacks reach high Bad Bot rates (e.g., GPT-4 ~0.93) and substantial Good Bot rates for defenses (varies by model).
• Adaptive attacks can achieve near-universal success across prompts for GPT-4 and Claude v1.3 in the curated dataset.
• Safety improvements do not fully prevent jailbreaks; scale alone does not resolve failure modes and may shift vulnerability surfaces.
• Safety-capability parity is argued as necessary to defend against adversarial use of LLMs.