[Paper Review] FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models

Dongyu Yao, Jianshu Zhang+2 more|arXiv (Cornell University)|Sep 11, 2023

Adversarial Robustness in Machine Learning23 references4 citations

TL;DR

FuzzLLM is a novel, universal fuzzing framework that proactively discovers jailbreak vulnerabilities in large language models (LLMs) by automatically generating diverse, structurally varied prompts using templates, constraints, and prohibited questions. It achieves high effectiveness—successfully identifying jailbreaks in state-of-the-art models like GPT-3.5-turbo and GPT-4—even with small test sets, outperforming single-component attacks in combo-attack scenarios.

ABSTRACT

Jailbreak vulnerabilities in Large Language Models (LLMs), which exploit meticulously crafted prompts to elicit content that violates service guidelines, have captured the attention of research communities. While model owners can defend against individual jailbreak prompts through safety training strategies, this relatively passive approach struggles to handle the broader category of similar jailbreaks. To tackle this issue, we introduce FuzzLLM, an automated fuzzing framework designed to proactively test and discover jailbreak vulnerabilities in LLMs. We utilize templates to capture the structural integrity of a prompt and isolate key features of a jailbreak class as constraints. By integrating different base classes into powerful combo attacks and varying the elements of constraints and prohibited questions, FuzzLLM enables efficient testing with reduced manual effort. Extensive experiments demonstrate FuzzLLM's effectiveness and comprehensiveness in vulnerability discovery across various LLMs.

Motivation & Objective

• To address the reactive nature of current LLM safety defenses, which patch individual jailbreaks after public disclosure, by enabling proactive vulnerability discovery.
• To overcome the scarcity of diverse, labeled jailbreak data that limits effective safety fine-tuning in LLMs.
• To develop a universal, automated framework that can test any LLM for jailbreak vulnerabilities without requiring access to model internals.
• To systematically generate varied, robust jailbreak prompts by decomposing attacks into reusable components: templates, constraints, and question sets.
• To evaluate the effectiveness of combo attacks that combine multiple jailbreak classes (Role Play, Output Constrain, Privilege Escalation) for broader vulnerability coverage.

Proposed method

• FuzzLLM uses black-box, IO-driven fuzzing to test LLMs without internal access, generating jailbreak prompts via template-based composition.
• It decomposes jailbreaks into three core components: a structural template, a constraint set (key features of successful jailbreaks), and a prohibited question set (policy-violating queries).
• The framework combines three base jailbreak classes—Role Play (RP), Output Constrain (OC), and Privilege Escalation (PE)—into powerful combo attacks by mixing constraints and questions.
• It employs a label model (e.g., Vicuna-13B) to automatically classify outputs as 'bad' (jailbroken) or 'good' (safe), with error rates measured to validate reliability.
• The system uses hyperparameters: test set size Tes=300 per class (2100 total), temperature=0.7, and max output token tk=256, with results averaged over three random seeds.
• Ablation studies assess the impact of test set size and output token limit, revealing minimal performance variation across sizes and a counterintuitive spike in success rate at low tk=64 due to incomplete output detection.

Experimental results

Research questions

• RQ1 Can a universal, automated fuzzing framework effectively discover jailbreak vulnerabilities across diverse LLMs without requiring model internals?
• RQ2 How does combining multiple base jailbreak classes (RP, OC, PE) into combo attacks improve vulnerability discovery compared to single-component attacks?
• RQ3 What is the impact of test set size and output token limit on the reliability and performance of jailbreak detection using automated labeling?
• RQ4 How effective is FuzzLLM in uncovering jailbreaks in state-of-the-art, well-protected LLMs like GPT-3.5-turbo and GPT-4?
• RQ5 Can a lightweight, open-sourced LLM serve as a reliable label model for classifying jailbreak success with acceptable error rates?

Key findings

• FuzzLLM successfully uncovers jailbreak vulnerabilities in GPT-3.5-turbo and GPT-4 despite their advanced safety defenses, demonstrating its proactive discovery capability.
• Combo attacks combining multiple base classes (RP, OC, PE) achieve a 19.61% jailbreak success rate on GPT-4—significantly higher than single-component attacks (11.92%)—indicating superior coverage and potency.
• The label model Vicuna-13B achieves the lowest error rate (4.08%) among tested models (Bloom-7B: 14.35%, LLAMA-7B: 11.57%), validating its reliability for automated labeling.
• Ablation studies show minimal performance variation across different test set sizes (Tes=50 to 500), indicating that results are stable and generalizable even with small-scale testing.
• The success rate spikes at tk=64 (82.26%) due to incomplete outputs being misclassified as 'bad' when only the malicious question is present, revealing a critical sensitivity in output length for detection.
• FuzzLLM outperforms single-component attacks on GPT-3.5-turbo (23.57% vs. 23.12%) and GPT-4 (19.61% vs. 11.92%) in combo-attack mode, confirming the effectiveness of combined attack patterns.