[Paper Review] Personalized Language Modeling from Personalized Human Feedback

Xinyu Li, Zhou, Ruiyang+2 more|arXiv (Cornell University)|Feb 6, 2024

Topic Modeling4 citations

TL;DR

This paper proposes Personalized-RLHF (P-RLHF), a framework that jointly learns a user model and a personalized language/reward model to align large language models with individual user preferences. By incorporating user-specific embeddings and novel learning objectives for personalized reward modeling (P-RM) and Direct Preference Optimization (P-DPO), the method improves alignment with both seen and unseen users, achieving up to 61.72% accuracy on personalized preference prediction—surpassing non-personalized baselines.

ABSTRACT

Personalized large language models (LLMs) are designed to tailor responses to individual user preferences. While Reinforcement Learning from Human Feedback (RLHF) is a commonly used framework for aligning LLMs with human preferences, vanilla RLHF assumes that all human preferences share the same distribution, preventing fine-tuned LLMs from generating personalized content when user preferences are diverse. In this work, we propose Personalized-RLHF (P-RLHF), an efficient framework that utilizes a lightweight user model to capture individual user preferences and jointly learns the user model and the personalized LLM from human feedback. P-RLHF exhibits the following three characteristics: (1) It enables an LLM to generate personalized content and scale efficiently with growing number of users. (2) It handles both explicit user preferences described as textual input and implicit user preferences encoded in the feedback data. (3) It eliminates the need for users to fully articulate their preferences, which are normally needed for prompting LLMs to generate personalized content yet are often impractical to obtain in real-world scenarios. Our experimental results show that personalized LLMs trained using P-RLHF generate responses that are more closely aligned with individual user preferences, outperforming vanilla, non-personalized RLHF and prompting-based personalization approaches across different tasks. We opensource our code at https://github.com/HumainLab/Personalized_RLHF.

Motivation & Objective

• To address the limitations of vanilla RLHF in handling diverse, personalized human preferences in large language models.
• To formalize the task of learning from personalized human feedback, where user preferences vary significantly across individuals.
• To develop a general P-RLHF framework that jointly learns user representations and personalized reward/language models.
• To design new learning objectives for personalized reward modeling (P-RM) and personalized Direct Preference Optimization (P-DPO).
• To evaluate the framework on real-world text summarization data with annotated user preferences, demonstrating improved alignment with individual users.

Proposed method

• Proposes a P-RLHF framework that jointly trains a user model and a language/reward model using preference data annotated with user identities.
• Designs a user model that maps user information (e.g., worker IDs) into user-specific embeddings, with options for individual or cluster-based representations.
• Introduces a personalized reward modeling objective (P-RM) that combines user-specific and user-agnostic loss terms, weighted by a hyperparameter α.
• Develops a personalized DPO objective (P-DPO) that optimizes the language model using a user-conditioned implicit reward function.
• Uses a hybrid loss in P-RM and P-DPO that balances individual preference fitting with regularization via a generic user representation (e₀) for unseen users.
• Employs GPT-J 6B as the base language model and fine-tunes it using P-RM and P-DPO on a real-world text summarization dataset with 10 annotators.

Experimental results

Research questions

• RQ1 Can a unified framework jointly learn user representations and personalized reward models to improve alignment with individual preferences?
• RQ2 How does incorporating user-specific embeddings affect performance compared to non-personalized baselines in preference modeling?
• RQ3 What is the impact of the user-agnostic regularization term (α) in the P-RM and P-DPO objectives on generalization to unseen users?
• RQ4 How do different user model designs (individual vs. clustered) affect personalization performance and robustness?
• RQ5 Can the proposed method effectively tailor language model outputs to unseen users using a generic user representation?

Key findings

• The P-RM model with individual user embeddings and α=0.5 achieved 61.72% accuracy on top-1 preference predictions for seen users, outperforming the vanilla RM (60.27%).
• The P-DPO model with individual user embeddings and α=0.5 achieved 61.33% accuracy on seen users and 61.97% on unseen users, surpassing the vanilla DPO baseline (60.48% and 60.99%, respectively).
• The user-agnostic loss term (α=0.5) significantly improved generalization, as models with α=0.5 outperformed those with α=1.0 across all evaluation metrics.
• Cluster-based user models with K=5 performed comparably to individual models but showed higher variance, suggesting potential for improvement with better hyperparameter tuning or domain-aware clustering.
• The generic user representation (e₀) enabled effective personalization for unseen users, with P-DPO achieving 61.97% accuracy on unseen users—higher than the vanilla DPO baseline.
• The results demonstrate that explicitly modeling user preferences through user representations leads to better alignment with individual preferences than standard RLHF approaches.