[Paper Review] Aligning LLM Agents by Learning Latent Preference from User Edits

Ge Gao, Alexey Taymanov+3 more|arXiv (Cornell University)|Apr 23, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes PRELUDE, a framework that learns latent user preferences from natural user edits to align frozen LLM agents without fine-tuning. Using the CIPHER algorithm, it infers preference descriptions via LLMs, retrieves similar historical edits, and aggregates them into a compact prompt policy—achieving lower edit distance and computational cost than baselines while improving interpretability.

ABSTRACT

We study interactive learning of LLM-based language agents based on user edits made to the agent's output. In a typical setting such as writing assistants, the user interacts with a language agent to generate a response given a context, and may optionally edit the agent response to personalize it based on their latent preference, in addition to improving the correctness. The edit feedback is naturally generated, making it a suitable candidate for improving the agent's alignment with the user's preference, and for reducing the cost of user edits over time. We propose a learning framework, PRELUDE that infers a description of the user's latent preference based on historic edit data. The inferred user preference descriptions are used to define prompts for generating responses in the future. This avoids fine-tuning the agent, which is costly, challenging to scale with the number of users, and may even degrade its performance on other tasks. Furthermore, learning descriptive preference improves interpretability, allowing the user to view and modify the learned preference. However, user preference can be complex, subtle, and vary based on context, making it challenging to learn. To address this, we propose a simple yet effective algorithm named CIPHER that leverages the LLM to infer the user preference for a given context based on user edits. In the future, CIPHER retrieves inferred preferences from the k-closest contexts in the history, and forms an aggregate preference for response generation. We introduce two interactive environments -- summarization and email writing, and use a GPT-4 simulated user for evaluation. On both tasks, CIPHER outperforms several baselines by achieving the lowest edit distance cost while only having a small overhead in LLM query cost. Our analysis reports that user preferences learned by CIPHER show significant similarity to the ground truth latent preferences.

Motivation & Objective

• To reduce user interaction cost in LLM-based writing assistants by learning from natural, implicit edits.
• To model user preferences as describable, context-sensitive textual descriptions rather than binary comparisons.
• To avoid expensive fine-tuning by learning a prompt policy from edit history instead of retraining the model.
• To improve interpretability by exposing learned preferences to users for inspection and modification.
• To enable scalable personalization across users without per-user model adaptation.

Proposed method

• Propose PRELUDE, a framework that learns user preference descriptions from user edits to guide prompt policy generation.
• Introduce CIPHER, a retrieval-based algorithm that uses an LLM to infer preference descriptions from historical edits.
• Retrieve the k most similar contexts from history based on embedding distance to the current context.
• Aggregate the inferred preferences from the k nearest neighbors into a single, compact prompt for response generation.
• Use the aggregated preference description as a dynamic prompt to steer the frozen LLM agent’s response.
• Evaluate using two interactive environments (summarization and email writing) with a GPT-4 simulated user to simulate real-world edit feedback.

Figure 1: Illustration of interactive learning from user edits. Color coding in edits is for visualization only – our agent takes the plain revised text as feedback.

Experimental results

Research questions

• RQ1 Can user edits be effectively modeled as latent, describable preferences that generalize across contexts?
• RQ2 Can a retrieval-based method using inferred preference descriptions outperform direct edit retrieval or context-agnostic preference learning?
• RQ3 Does learning preference descriptions reduce user edit cost (e.g., edit distance) compared to baselines?
• RQ4 Can the learned preference be interpreted and modified by users, enhancing transparency?
• RQ5 Is the computational cost of using learned preferences lower than using raw edit sequences in prompts?

Key findings

• CIPHER achieved the lowest edit distance cost on both summarization and email writing tasks, outperforming baselines that directly retrieve edits or learn context-agnostic preferences.
• The use of learned preference descriptions reduced prompt length compared to raw edit sequences, lowering computational cost.
• The inferred preferences showed significant similarity to ground-truth latent preferences, indicating accurate preference modeling.
• The framework enables interpretable alignment, as users can view and modify the learned preference description.
• The method scales efficiently across users, avoiding the need for per-user fine-tuning.
• The GPT-4 simulated user evaluation demonstrated consistent performance gains across diverse interaction scenarios.