[Paper Review] Fundamental Limitations of Alignment in Large Language Models

Yotam Wolf, Noam Wies+3 more|arXiv (Cornell University)|Apr 19, 2023

Topic ModelingComputer Science43 citations

TL;DR

The paper introduces Behavior Expectation Bounds (BEB), a probabilistic framework showing that any aligned LLM can be adversarially misaligned by prompts, and that preset prompts or multi-turn conversations do not fully prevent misalignment.

ABSTRACT

An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users. This is usually achieved by tuning the model in a way that enhances desired behaviors and inhibits undesired ones, a process referred to as alignment. In this paper, we propose a theoretical approach called Behavior Expectation Bounds (BEB) which allows us to formally investigate several inherent characteristics and limitations of alignment in large language models. Importantly, we prove that within the limits of this framework, for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt. This implies that any alignment process that attenuates an undesired behavior but does not remove it altogether, is not safe against adversarial prompting attacks. Furthermore, our framework hints at the mechanism by which leading alignment approaches such as reinforcement learning from human feedback make the LLM prone to being prompted into the undesired behaviors. This theoretical result is being experimentally demonstrated in large scale by the so called contemporary "chatGPT jailbreaks", where adversarial users trick the LLM into breaking its alignment guardrails by triggering it into acting as a malicious persona. Our results expose fundamental limitations in alignment of LLMs and bring to the forefront the need to devise reliable mechanisms for ensuring AI safety.

Motivation & Objective

• Formalize the limitations of aligning large language models (LLMs) using a probabilistic framework.
• Show that alignment that reduces but does not remove undesired behavior remains vulnerable to adversarial prompting.
• Analyze how prompting, preset aligning prompts, and conversations influence misalignment risk.
• Provide empirical support on LLaMA families to illustrate BEB parameters and misalignment dynamics.

Proposed method

• Define Behavior Expectation Bounds (BEB) to quantify expected behavior scores along behavior verticals.
• Model LLM distributions as mixtures of ill-behaved and well-behaved components and study prompt reweighting of these components.
• Introduce alpha, beta, gamma-distinguishability and sigma-similarity to bound misalignment risk under prompting.
• Prove theorems showing misalignment via adversarial prompts (Theorem 1), misalignment with preset aligning prompts (Theorem 2), and misalignment in conversational settings (Theorem 3).
• Provide empirical estimates of BEB parameters (beta, beta', sigma) using LLaMA-2 models and Perez et al. behavior datasets.

Figure 1: Examples of sentence behavior scores along different behavior verticals. Our framework of Behavior Expectation Bounds (BEB) assumes ground truth behavior scoring functions, and bounds the expected scores of models along different behavior verticals in order to guarantee LLM alignment or mi

Experimental results

Research questions

• RQ1 Can an aligned LLM be provably misaligned by adversarial prompts under BEB assumptions?
• RQ2 How do preset aligning prompts affect the length of misalignment prompts required?
• RQ3 Does misalignment extend to multi-turn conversations, and under what prompt lengths?
• RQ4 What are realistic ranges for BEB parameters (beta, beta', sigma) in practical LLMs?
• RQ5 Do RLHF-tuned models exhibit dynamics consistent with BEB predictions?

Key findings

• Alignment is shown to be inherently vulnerable to adversarial prompts under alpha,beta,gamma-negatively-distinguishable distributions.
• An aligning prefix does not guarantee safety; the required misaligning prompt length grows with the aligning prompt length.
• Misalignment can occur during conversations, with prompt length requirements accumulating over turns.
• Empirical estimates from LLaMA-2 indicate beta around 20, beta' around 30, and sigma^2 around 50 for agreeableness behavior, with sigma/beta about 0.35.
• RLHF-finetuned LLMs demonstrate convergence toward negative-behavior distributions under adversarial prompting, aligning with BEB predictions.

Figure 2: (a) KL between two distributions of opposite behaviors as function of prompt length sampled from $\mathbb{P}_{-}$ , averaged on 10 sampled sequences. For these two distributions, we see $\beta\approx 20$ . (b) Corresponding log ratio variance between the distributions mentioned in (a). 30