[论文解读] Aligning LLM Agents by Learning Latent Preference from User Edits

Ge Gao, Alexey Taymanov+3 more|arXiv (Cornell University)|Apr 23, 2024

Natural Language Processing Techniques被引用 4

一句话总结

该论文提出 PRELUDE，一种从自然用户编辑中学习潜在用户偏好的框架，无需微调即可对齐冻结的 LLM 代理。通过 CIPHER 算法，它利用 LLM 推断偏好描述，检索类似的过往编辑，并将它们聚合为紧凑的提示策略——在保持更低编辑距离和计算成本的同时，提升了可解释性。

ABSTRACT

We study interactive learning of LLM-based language agents based on user edits made to the agent's output. In a typical setting such as writing assistants, the user interacts with a language agent to generate a response given a context, and may optionally edit the agent response to personalize it based on their latent preference, in addition to improving the correctness. The edit feedback is naturally generated, making it a suitable candidate for improving the agent's alignment with the user's preference, and for reducing the cost of user edits over time. We propose a learning framework, PRELUDE that infers a description of the user's latent preference based on historic edit data. The inferred user preference descriptions are used to define prompts for generating responses in the future. This avoids fine-tuning the agent, which is costly, challenging to scale with the number of users, and may even degrade its performance on other tasks. Furthermore, learning descriptive preference improves interpretability, allowing the user to view and modify the learned preference. However, user preference can be complex, subtle, and vary based on context, making it challenging to learn. To address this, we propose a simple yet effective algorithm named CIPHER that leverages the LLM to infer the user preference for a given context based on user edits. In the future, CIPHER retrieves inferred preferences from the k-closest contexts in the history, and forms an aggregate preference for response generation. We introduce two interactive environments -- summarization and email writing, and use a GPT-4 simulated user for evaluation. On both tasks, CIPHER outperforms several baselines by achieving the lowest edit distance cost while only having a small overhead in LLM query cost. Our analysis reports that user preferences learned by CIPHER show significant similarity to the ground truth latent preferences.

研究动机与目标

• 通过从自然的、隐式的编辑中学习，减少基于 LLM 的写作助手中的用户交互成本。
• 将用户偏好建模为可描述的、上下文敏感的文本描述，而非二元比较。
• 通过从编辑历史中学习提示策略而非重新训练模型，避免昂贵的微调。
• 通过向用户暴露学习到的偏好以供检查和修改，提升可解释性。
• 在无需为每个用户单独调整模型的前提下，实现跨用户的可扩展个性化。

提出的方法

• 提出 PRELUDE 框架，从用户编辑中学习用户偏好描述，以指导提示策略的生成。
• 引入 CIPHER，一种基于检索的算法，利用 LLM 从历史编辑中推断偏好描述。
• 基于当前上下文与历史记录的嵌入距离，检索最相似的 k 个上下文。
• 将 k 个最近邻中推断出的偏好聚合为一个紧凑的单一提示，用于响应生成。
• 使用聚合后的偏好描述作为动态提示，引导冻结的 LLM 代理生成响应。
• 在两个交互式环境（摘要生成和邮件写作）中进行评估，使用 GPT-4 模拟用户以模拟真实世界的编辑反馈。

Figure 1: Illustration of interactive learning from user edits. Color coding in edits is for visualization only – our agent takes the plain revised text as feedback.

实验结果

研究问题

• RQ1 用户编辑能否被有效建模为可描述的、泛化于多种上下文的潜在偏好？
• RQ2 基于推断偏好描述的检索方法是否优于直接编辑检索或上下文无关的偏好学习？
• RQ3 与基线方法相比，学习偏好描述是否能降低用户编辑成本（如编辑距离）？
• RQ4 学习到的偏好是否可被用户理解并修改，从而增强透明度？
• RQ5 使用学习到的偏好相比在提示中使用原始编辑序列，其计算成本是否更低？

主要发现

• 在摘要生成和邮件写作任务中，CIPHER 均实现了最低的编辑距离成本，优于直接检索编辑或学习上下文无关偏好的基线方法。
• 使用学习到的偏好描述相比原始编辑序列，显著缩短了提示长度，降低了计算成本。
• 推断出的偏好与真实潜在偏好具有显著相似性，表明偏好建模准确。
• 该框架实现了可解释的对齐，用户可查看并修改学习到的偏好描述。
• 该方法在用户间具有高效的可扩展性，避免了为每个用户进行微调的需要。
• 使用 GPT-4 模拟用户的评估表明，该方法在多样化交互场景中均表现出一致的性能提升。