[Paper Review] A StrongREJECT for Empty Jailbreaks

Alexandra Souly, Qingyuan Lu+9 more|arXiv (Cornell University)|Feb 15, 2024

Digital and Cyber Forensics4 citations

TL;DR

This paper introduces StrongREJECT, a new benchmark to fairly evaluate jailbreak attacks on large language models by addressing flaws in existing evaluation methods. It proposes higher-quality, specific, and answerable harmful questions across six banned categories and a more accurate autograder that prioritizes response usefulness over mere toxicity or refusal failure, reducing bias in performance estimation and showing that some jailbreaks degrade model performance even on benign tasks.

ABSTRACT

Most jailbreak papers claim the jailbreaks they propose are highly effective, often boasting near-100% attack success rates. However, it is perhaps more common than not for jailbreak developers to substantially exaggerate the effectiveness of their jailbreaks. We suggest this problem arises because jailbreak researchers lack a standard, high-quality benchmark for evaluating jailbreak performance, leaving researchers to create their own. To create a benchmark, researchers must choose a dataset of forbidden prompts to which a victim model will respond, along with an evaluation method that scores the harmfulness of the victim model's responses. We show that existing benchmarks suffer from significant shortcomings and introduce the StrongREJECT benchmark to address these issues. StrongREJECT's dataset contains prompts that victim models must answer with specific, harmful information, while its automated evaluator measures the extent to which a response gives useful information to forbidden prompts. In doing so, the StrongREJECT evaluator achieves state-of-the-art agreement with human judgments of jailbreak effectiveness. Notably, we find that existing evaluation methods significantly overstate jailbreak effectiveness compared to human judgments and the StrongREJECT evaluator. We describe a surprising and novel phenomenon that explains this discrepancy: jailbreaks bypassing a victim model's safety fine-tuning tend to reduce its capabilities. Together, our findings underscore the need for researchers to use a high-quality benchmark, such as StrongREJECT, when developing new jailbreak attacks. We release the StrongREJECT code and data at https://strong-reject.readthedocs.io/en/latest/.

Motivation & Objective

• To identify and address critical flaws in existing jailbreak evaluation benchmarks that overestimate attack success due to poor question quality and biased grading.
• To demonstrate that some jailbreaking techniques degrade model performance on benign tasks, such as reducing GPT-4’s MMLU zero-shot accuracy from 78% to 35%.
• To propose a new benchmark, StrongREJECT, that improves evaluation fairness by using specific, answerable, and harmful questions across six content categories.
• To develop a response grading algorithm that better aligns with human judgment by emphasizing the usefulness of responses in achieving harmful goals, not just refusal or toxicity.
• To provide a more accurate and reliable standard for measuring jailbreak effectiveness, reducing overestimation of attack success due to low-quality responses.

Proposed method

• Design a new question set from scratch and curated from existing datasets, ensuring all questions are specific, answerable, and fall into six widely prohibited content categories.
• Construct a new autograder that evaluates jailbreak responses based on their usefulness in achieving a harmful objective, rather than just detecting toxicity or refusal failure.
• Use human-annotated judgments to calibrate and validate the new grading algorithm, ensuring it aligns closely with human perception of response quality and attack effectiveness.
• Apply the new benchmark to evaluate multiple jailbreak methods on both closed-source (GPT-4) and uncensored open-source models (Dolphin), measuring performance degradation and attack success rates.
• Implement and release the full benchmark, including questions, grading code, and evaluation results, via a public GitHub repository for reproducibility and community use.
• Use bootstrap confidence intervals to assess statistical significance in performance differences across jailbreak methods on the new benchmark.

Figure 1 : MMLU zero-shot accuracy using GPT-4 after applying jailbreaks to the MMLU prompt. Accuracy is calculated over valid responses only. Most of these jailbreaks decrease accuracy.

Experimental results

Research questions

• RQ1 To what extent do existing jailbreak benchmarks overestimate attack success due to vague, unanswerable, or low-quality questions?
• RQ2 How do different jailbreak techniques affect the performance of LLMs on benign, non-malicious benchmarks like MMLU?
• RQ3 To what extent does the current autograding methodology in existing benchmarks bias results toward overestimating the effectiveness of low-quality jailbreak responses?
• RQ4 How well does the proposed StrongREJECT autograder align with human judgment in evaluating both individual response quality and overall jailbreak effectiveness?
• RQ5 Can a new benchmark with higher-quality questions and a more accurate grading scheme provide a more balanced and reliable assessment of jailbreak capabilities?

Key findings

• Several jailbreak methods significantly reduce GPT-4’s zero-shot accuracy on the MMLU benchmark, dropping from 78% to as low as 35%, even when only counting valid multiple-choice responses.
• On the uncensored Dolphin model, certain jailbreaks—such as Base64 obfuscation and Zulu language translation—decrease the model’s ability to correctly answer harmful questions, indicating performance degradation.
• The proposed StrongREJECT autograder shows strong alignment with human judgment in evaluating both individual response quality and overall jailbreak effectiveness, especially for low-quality responses.
• Existing autograders often overestimate jailbreak success by giving credit for responses that merely contain toxic content or fail to refuse a request explicitly, leading to inflated performance scores.
• The new benchmark successfully identifies that some jailbreaks are ineffective or harmful to model capability, revealing that not all jailbreaks improve misuse potential.
• The StrongREJECT benchmark, with its improved question set and grading algorithm, provides a more balanced and accurate assessment of jailbreak effectiveness than existing benchmarks.

Figure 2 : StrongREJECT improves on existing jailbreak benchmarks by asking questions that are specific, answerable, and harmful across six categories of content that are widely rejected by models. StrongREJECT also uses an autograder that emphasizes the usefulness of a response for achieving a part