[论文解读] Fundamental Limitations of Alignment in Large Language Models

Yotam Wolf, Noam Wies+3 more|arXiv (Cornell University)|Apr 19, 2023

Topic Modeling被引用 43

一句话总结

本文提出 Behavior Expectation Bounds (BEB)，这是一种概率框架，表明任何对齐的大语言模型都可能被提示词有 adversarial misalignment，通过提示进行对抗性错位，且预设提示词或多轮对话并不能完全防止错位。

ABSTRACT

An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users. This is usually achieved by tuning the model in a way that enhances desired behaviors and inhibits undesired ones, a process referred to as alignment. In this paper, we propose a theoretical approach called Behavior Expectation Bounds (BEB) which allows us to formally investigate several inherent characteristics and limitations of alignment in large language models. Importantly, we prove that within the limits of this framework, for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt. This implies that any alignment process that attenuates an undesired behavior but does not remove it altogether, is not safe against adversarial prompting attacks. Furthermore, our framework hints at the mechanism by which leading alignment approaches such as reinforcement learning from human feedback make the LLM prone to being prompted into the undesired behaviors. This theoretical result is being experimentally demonstrated in large scale by the so called contemporary "chatGPT jailbreaks", where adversarial users trick the LLM into breaking its alignment guardrails by triggering it into acting as a malicious persona. Our results expose fundamental limitations in alignment of LLMs and bring to the forefront the need to devise reliable mechanisms for ensuring AI safety.

研究动机与目标

• 用概率框架形式化对齐大语言模型（LLMs）的局限性。
• 表明降低但未消除不良行为的对齐仍易受到对抗性提示的影响。
• 分析提示、预设对齐提示和对话如何影响错位风险。
• 提供关于 LLaMA 系列的实证支持，以说明 BEB 参数和错位动态。

提出的方法

• 定义 Behavior Expectation Bounds (BEB)，以在行为纵轴上量化期望行为分数。
• 将 LLM 分布建模为不良行为与良好行为组件的混合，并研究这些组件的提示重新加权。
• 引入 alpha、beta、gamma 可辨别性和 sigma 相似性，以在提示下对错位风险进行界定。
• 证明定理1：通过对抗性提示导致错位；定理2：存在使用预设对齐提示时的错位；定理3：在对话情境下的错位。
• 使用 LLaMA-2 模型与 Perez 等人的行为数据集，提供 BEB 参数（beta、beta'、sigma）的实证估计。

Figure 1: Examples of sentence behavior scores along different behavior verticals. Our framework of Behavior Expectation Bounds (BEB) assumes ground truth behavior scoring functions, and bounds the expected scores of models along different behavior verticals in order to guarantee LLM alignment or mi

实验结果

研究问题

• RQ1 在 BEB 假设下，是否可以对齐的 LLM 被对抗性提示明确地错位？
• RQ2 预设对齐提示如何影响所需错位提示的长度？
• RQ3 错位是否会扩展到多轮对话，在何种提示长度下？
• RQ4 在实际的 LLM 中，BEB 参数（beta、beta'、sigma）的现实区间是多少？
• RQ5 经 RLHF 调整的模型是否表现出与 BEB 预测一致的动态？

主要发现

• 在 alpha、beta、gamma 可区分为负的分布下，对齐被对抗性提示本质上容易受到攻击。
• 对齐前缀并不能保证安全；所需的错位提示长度随对齐提示长度增加而增加。
• 对话中也可能发生错位，提示长度的需求会在轮次中累积。
• 来自 LLaMA-2 的实证估计表明，顺从行为的 beta 约为 20，beta' 约为 30，sigma^2 约为 50，且 sigma/beta 约为 0.35。
• RLHF 微调的 LLM 在对抗性提示下表现出向负向行为分布收敛，与 BEB 的预测一致。

Figure 2: (a) KL between two distributions of opposite behaviors as function of prompt length sampled from $\mathbb{P}_{-}$ , averaged on 10 sampled sequences. For these two distributions, we see $\beta\approx 20$ . (b) Corresponding log ratio variance between the distributions mentioned in (a). 30