[Paper Review] A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily

Peng Ding, J. S. Kuang+5 more|arXiv (Cornell University)|Nov 14, 2023

Computational and Text Analysis Methods4 citations

TL;DR

This paper proposes ReNeLLM, an automated framework that generalizes jailbreak prompt attacks through prompt rewriting and scenario nesting to bypass LLM safety alignments. By leveraging LLMs to generate semantically preserved yet stealthier prompts embedded in common tasks, ReNeLLM achieves over 40% higher attack success rates than baselines while reducing time costs, exposing critical vulnerabilities in current defense mechanisms.

ABSTRACT

Large Language Models (LLMs), such as ChatGPT and GPT-4, are designed to provide useful and safe responses. However, adversarial prompts known as 'jailbreaks' can circumvent safeguards, leading LLMs to generate potentially harmful content. Exploring jailbreak prompts can help to better reveal the weaknesses of LLMs and further steer us to secure them. Unfortunately, existing jailbreak methods either suffer from intricate manual design or require optimization on other white-box models, which compromises either generalization or efficiency. In this paper, we generalize jailbreak prompt attacks into two aspects: (1) Prompt Rewriting and (2) Scenario Nesting. Based on this, we propose ReNeLLM, an automatic framework that leverages LLMs themselves to generate effective jailbreak prompts. Extensive experiments demonstrate that ReNeLLM significantly improves the attack success rate while greatly reducing the time cost compared to existing baselines. Our study also reveals the inadequacy of current defense methods in safeguarding LLMs. Finally, we analyze the failure of LLMs defense from the perspective of prompt execution priority, and propose corresponding defense strategies. We hope that our research can catalyze both the academic community and LLMs developers towards the provision of safer and more regulated LLMs. The code is available at https://github.com/NJUNLP/ReNeLLM.

Motivation & Objective

• To address the limitations of manual and optimization-based jailbreak methods, which are time-consuming, fragile, or semantically unnatural.
• To generalize jailbreak attacks by identifying two core patterns: prompt rewriting and scenario nesting.
• To develop an automated, efficient, and effective framework that uses LLMs themselves to generate high-success-rate jailbreak prompts without fine-tuning or optimization.
• To evaluate the robustness of current defense mechanisms against these generalized attacks.
• To provide insights into why LLMs fail to detect malicious content by analyzing attention mechanisms during prompt execution.

Proposed method

• ReNeLLM generalizes jailbreak attacks into two components: prompt rewriting and scenario nesting.
• Prompt rewriting applies linguistic transformations (e.g., synonym substitution, syntax rephrasing, misspellings) to preserve core semantics while evading detection.
• Scenario nesting embeds rewritten prompts within common task contexts such as table filling, text completion, or code generation to mask malicious intent.
• The framework uses LLMs to automatically explore and select the optimal combination of rewriting and nesting strategies.
• It operates without additional training or gradient-based optimization, relying solely on LLMs’ reasoning and generation capabilities.
• Attention visualization is used to analyze how LLMs prioritize external vs. internal instructions during execution, revealing shifts in attention that enable evasion.

Figure 1: An example of jailbreak prompt given by ReNeLLM.

Experimental results

Research questions

• RQ1 Can generalized patterns of prompt rewriting and scenario nesting significantly improve jailbreak success rates across diverse LLMs?
• RQ2 How does the combination of semantic-preserving rewriting and contextual nesting affect LLMs’ attention and response behavior?
• RQ3 Why do current defense mechanisms fail to block ReNeLLM-generated jailbreak prompts despite their semantic coherence?
• RQ4 To what extent does ReNeLLM outperform existing baselines in attack success rate and efficiency?
• RQ5 How do attention mechanisms in LLMs shift during execution of nested jailbreak prompts, and what does this imply for model safety?

Key findings

• ReNeLLM achieves over 40% higher attack success rate on Llama-2-chat-70b compared to baseline methods, demonstrating significant performance gains.
• The attack success rate on GPT-3.5 increases by over 40% when combining prompt rewriting and scenario nesting, indicating strong synergy between the two components.
• Attention visualization shows that after nesting, LLMs prioritize external task instructions over internal malicious prompts, reducing detection likelihood.
• Defenses based on perplexity or semantic filtering fail against ReNeLLM, as rewritten prompts maintain high fluency and coherence.
• The framework reduces time cost significantly compared to optimization-based methods like GCG, which require iterative search and gradient computation.
• Even with safety-aligned fine-tuning, LLMs remain vulnerable to ReNeLLM’s generalized attack pattern, exposing fundamental weaknesses in current alignment techniques.

Figure 2: Overview of ReNeLLM, an automatic jailbreak prompt generation framework. For the first time, it generalizes jailbreak prompt attacks, including two aspects, prompt rewriting and scenario nesting. The former is inspired by linguistics Chomsky ( 2002 ) , performing various rewriting operatio