[Paper Review] Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models

Huachuan Qiu, Shuai Zhang+3 more|arXiv (Cornell University)|Jul 17, 2023

Topic Modeling11 citations

TL;DR

The paper introduces Latent Jailbreak, a dataset and framework to evaluate both safety and output robustness of LLMs against latent jailbreak prompts, analyzing the impact of instruction positioning, cue words, target groups, verbs, and toxic adjectives on jailbreak success and robustness.

ABSTRACT

Considerable research efforts have been devoted to ensuring that large language models (LLMs) align with human values and generate safe text. However, an excessive focus on sensitivity to certain topics can compromise the model's robustness in following instructions, thereby impacting its overall performance in completing tasks. Previous benchmarks for jailbreaking LLMs have primarily focused on evaluating the safety of the models without considering their robustness. In this paper, we propose a benchmark that assesses both the safety and robustness of LLMs, emphasizing the need for a balanced approach. To comprehensively study text safety and output robustness, we introduce a latent jailbreak prompt dataset, each involving malicious instruction embedding. Specifically, we instruct the model to complete a regular task, such as translation, with the text to be translated containing malicious instructions. To further analyze safety and robustness, we design a hierarchical annotation framework. We present a systematic analysis of the safety and robustness of LLMs regarding the position of explicit normal instructions, word replacements (verbs in explicit normal instructions, target groups in malicious instructions, cue words for explicit normal instructions), and instruction replacements (different explicit normal instructions). Our results demonstrate that current LLMs not only prioritize certain instruction verbs but also exhibit varying jailbreak rates for different instruction verbs in explicit normal instructions. Code and data are available at https://github.com/qiuhuachuan/latent-jailbreak.

Motivation & Objective

• Motivate a balanced evaluation of safety and robustness in LLMs beyond jailbreak success alone.
• Propose latent jailbreak prompts by embedding malicious instructions into regular tasks (e.g., translation).
• Develop a hierarchical annotation framework to assess text safety and output robustness under various prompt manipulations.

Proposed method

• Construct a latent jailbreak prompt dataset with explicit normal instructions and implicit malicious instructions, including positional variants (prefix/suffix).
• Develop 13 prompt templates and 416 total latent jailbreak prompts across 16 target-group words and 2 positions.
• Design a Safety and Robustness Framework with binary safety labeling and robustness assessment linked to explicit normal instructions.
• Use three Chinese-English capable LLMs (ChatGLM2-6B, BELLE-7B-2M, ChatGPT) with nucleus sampling and fixed hyperparameters to generate 30 outputs per prompt.
• Fine-tune a RoBERTa classifier on human-labeled data to automatically label model outputs for safety and robustness.
• Evaluate jailbreak success rate P_J, instruction-following rate P_R, and a combined trustworthiness P = (1 - P_J) * P_R.

Experimental results

Research questions

• RQ1 How do latent jailbreak prompts affect the safety of LLM outputs across models?
• RQ2 How does the position of explicit normal instructions (prefix vs suffix) influence safety and robustness?
• RQ3 What is the effect of cue words, target groups, instruction verbs, and toxic adjectives on jailbreak susceptibility and instruction-following robustness?
• RQ4 Can automatic labeling via a fine-tuned RoBERTa accurately predict safety and robustness of model outputs across prompts?

Key findings

• ChatGLM2-6B shows the highest latent jailbreak vulnerability (75.3% for P1).
• BELLE-7B-2M shows moderate vulnerability (50.4% for P1) and strong instruction-following in several settings.
• ChatGPT yields fewer unsafe responses but struggles with following explicit instructions, showing lower robustness in some prompts.
• Cue words like 'sentence' generally reduce jailbreak success across models.
• Verbs in explicit instructions influence safety, with some verbs increasing unsafe outputs in certain models.
• Toxic adjectives in implicit instructions elicit varying unsafe responses across models, with BELLE-7B-2M showing sensitivity to certain adjectives.