[Paper Review] Intention Analysis Makes LLMs A Good Jailbreak Defender

Yuqi Zhang, Liang Ding+2 more|arXiv (Cornell University)|Jan 12, 2024

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes Intention Analysis Prompting (IAPrompt), a novel, inference-only defense that enhances LLM safety by guiding models to first analyze the user's underlying intention before generating responses. By leveraging a two-stage process—intention analysis followed by policy-aligned response—IAPrompt reduces jailbreak attack success rates by an average of 46.5% across multiple models and benchmarks, while preserving helpfulness.

ABSTRACT

Aligning large language models (LLMs) with human values, particularly when facing complex and stealthy jailbreak attacks, presents a formidable challenge. Unfortunately, existing methods often overlook this intrinsic nature of jailbreaks, which limits their effectiveness in such complex scenarios. In this study, we present a simple yet highly effective defense strategy, i.e., Intention Analysis ($\mathbb{IA}$). $\mathbb{IA}$ works by triggering LLMs' inherent self-correct and improve ability through a two-stage process: 1) analyzing the essential intention of the user input, and 2) providing final policy-aligned responses based on the first round conversation. Notably, $\mathbb{IA}$ is an inference-only method, thus could enhance LLM safety without compromising their helpfulness. Extensive experiments on varying jailbreak benchmarks across a wide range of LLMs show that $\mathbb{IA}$ could consistently and significantly reduce the harmfulness in responses (averagely -48.2% attack success rate). Encouragingly, with our $\mathbb{IA}$, Vicuna-7B even outperforms GPT-3.5 regarding attack success rate. We empirically demonstrate that, to some extent, $\mathbb{IA}$ is robust to errors in generated intentions. Further analyses reveal the underlying principle of $\mathbb{IA}$: suppressing LLM's tendency to follow jailbreak prompts, thereby enhancing safety.

Motivation & Objective

• To address the growing threat of stealthy jailbreak attacks that bypass traditional alignment methods in LLMs.
• To develop a defense mechanism that enhances safety without compromising model helpfulness or requiring fine-tuning.
• To leverage LLMs' intrinsic self-correction and intention understanding abilities for robust, scalable safety.
• To validate the effectiveness of intention analysis as a core component in detecting and mitigating malicious intent in user queries.

Proposed method

• IAPrompt employs a two-stage prompting framework: first, the model analyzes the essential intention of the user query with a focus on safety, ethics, and legality.
• The second stage generates a response aligned with safety policies, based on the analyzed intention from the first stage.
• The method uses Chain-of-Thought (CoT) prompting to guide step-by-step reasoning, enhancing interpretability and detection of hidden malicious intent.
• The approach is entirely inference-based, requiring no model fine-tuning or external tools, making it cost-effective and deployable immediately.
• It relies on the LLM’s own reasoning capability to detect subtle, stealthy jailbreak prompts by focusing on intent rather than surface-level tokens.
• The method is evaluated across diverse models (Vicuna, ChatGLM, MPT, DeepSeek, GPT-3.5) and benchmarks (SAP200, DAN, AdvBench), demonstrating consistent performance.

Experimental results

Research questions

• RQ1 Can LLMs effectively detect malicious intent in jailbreak prompts through self-analysis without external supervision?
• RQ2 Does a two-stage intention analysis and policy-aligned response mechanism significantly reduce attack success rates across diverse models and attack types?
• RQ3 How does the scale of the intention analysis model affect performance, and is a smaller model sufficient for effective detection?
• RQ4 To what extent does IAPrompt preserve model helpfulness while enhancing safety, especially on general-purpose benchmarks?
• RQ5 How well does LLM-based harm detection correlate with human-annotated harm levels?

Key findings

• IAPrompt reduces the average attack success rate (ASR) across SAP200 and DAN benchmarks by 46.5%, demonstrating strong and consistent defense performance.
• On the SAP200 and DAN datasets, IAPrompt achieves near-zero harmful response rates, with attack success rates dropping to 0.31% and 0.62% respectively for Vicuna-13B and Vicuna-7B models.
• The method maintains high helpfulness, as confirmed by TruthfulQA results, showing no degradation in general performance.
• Models like Vicuna-7B, Vicuna-13B, and MPT-30B-Chat achieve 100% success in intention analysis, indicating strong capability to detect hidden malicious intent.
• When the intention is correctly identified, 98.74% of responses are harmless, confirming the method’s strong alignment with safety policies.
• Human evaluation shows 97% agreement between annotators, and GPT-3.5 agreement with human labels reaches 99% for vanilla models and 93% for IAPrompt, indicating high reliability.