[Paper Review] Coercing LLMs to do and reveal (almost) anything

Jonas Geiping, Alex Stein+4 more|arXiv (Cornell University)|Feb 21, 2024

Law, AI, and Intellectual Property4 citations

TL;DR

This paper demonstrates that large language models (LLMs) can be coerced into unintended behaviors—such as misdirection, denial-of-service, data extraction, and role-hacking—through adversarial prompts that exploit their pre-training on code and residual 'glitch' tokens. The authors show that even RLHF-aligned models can be manipulated into harmful outputs, revealing that free-text input renders LLM outputs inherently insecure from a security standpoint.

ABSTRACT

It has recently been shown that adversarial attacks on large language models (LLMs) can "jailbreak" the model into making harmful statements. In this work, we argue that the spectrum of adversarial attacks on LLMs is much larger than merely jailbreaking. We provide a broad overview of possible attack surfaces and attack goals. Based on a series of concrete examples, we discuss, categorize and systematize attacks that coerce varied unintended behaviors, such as misdirection, model control, denial-of-service, or data extraction. We analyze these attacks in controlled experiments, and find that many of them stem from the practice of pre-training LLMs with coding capabilities, as well as the continued existence of strange "glitch" tokens in common LLM vocabularies that should be removed for security reasons.

Motivation & Objective

• To investigate the broader spectrum of adversarial attacks on LLMs beyond jailbreaking.
• To identify and categorize unintended behaviors induced by adversarial prompts, including misdirection, model control, and data leakage.
• To analyze the root causes of these vulnerabilities, particularly pre-training on code and residual 'glitch' tokens in vocabularies.
• To demonstrate that even RLHF-aligned models can be coerced into harmful outputs via free-text input.
• To argue that current LLM outputs cannot be assumed secure due to the ease of adversarial manipulation.

Proposed method

• The authors use prompt optimization techniques, including GCG (Greedy Coordinate Search), to generate adversarial inputs that coerce LLMs into unintended behaviors.
• They explore multiple attack objectives: misdirection (e.g., forcing model to respond in gibberish that triggers specific outputs), model control (e.g., inducing self-contradiction), and denial-of-service (e.g., triggering infinite loops).
• A novel attack type, 'collision attacks,' is introduced, where adversarial tokens are optimized to collide with target tokens (e.g., EOS or formatting tokens) to disrupt model behavior.
• The study analyzes model behavior in controlled experiments, observing self-contradictory responses and role-hacking phenomena where models comment on their own harmful outputs.
• The authors examine the role of 'glitch' tokens—rare, anomalous tokens in LLM vocabularies—that can be exploited to bypass safety mechanisms.
• They investigate the model's tendency to simulate code execution, showing that adversarial prompts often trigger code-like reasoning even when not intended.

Experimental results

Research questions

• RQ1 What are the full spectrum of unintended behaviors that can be coerced in LLMs through adversarial prompting?
• RQ2 Why are LLMs vulnerable to adversarial attacks that go beyond jailbreaking, such as misdirection or denial-of-service?
• RQ3 How do pre-training on code and residual 'glitch' tokens contribute to security vulnerabilities in LLMs?
• RQ4 To what extent can models recover from harmful outputs, and what does this reveal about their self-awareness or consistency?
• RQ5 Can adversarial attacks be systematically categorized, and what are the underlying mechanisms driving their success?

Key findings

• Adversarial attacks on LLMs extend far beyond jailbreaking, including misdirection, denial-of-service, and data extraction, demonstrating broader security risks.
• Many attacks succeed because models are pre-trained on code, leading them to simulate code execution even when prompted with non-code inputs.
• Residual 'glitch' tokens in LLM vocabularies—intended to be removed—can be exploited to bypass safety mechanisms and trigger unintended behaviors.
• Models often fail to maintain consistency, producing harmful outputs and then contradicting them with follow-up statements like 'Please answer without prohibited words,' indicating self-awareness issues.
• The study shows that even RLHF-aligned models like LLaMA2-7b-chat can be coerced into generating harmful content through carefully crafted adversarial prompts.
• The authors conclude that any LLM output generated from free-text user input must be assumed insecure, as the model can be manipulated into almost any behavior.