[Paper Review] Aligning to Thousands of Preferences via System Message Generalization

Seongyun Lee, Sue Hyun Park+2 more|arXiv (Cornell University)|May 28, 2024

Semantic Web and Ontologies4 citations

TL;DR

This paper proposes a scalable paradigm for aligning large language models (LLMs) to individual user preferences by training them to generalize across thousands of diverse system messages. Using a new 192k-sample dataset called the Multifaceted Collection, the authors train a 7B LLM named Janus, which achieves state-of-the-art performance on multiple benchmarks—outperforming Mistral 7B, GPT-3.5, and even LLaMA 3 8B Instruct—while maintaining strong alignment to unseen user preferences without retraining.

ABSTRACT

Although humans inherently have diverse values, current large language model (LLM) alignment methods often assume that aligning LLMs with the general public's preferences is optimal. A major challenge in adopting a more individualized approach to LLM alignment is its lack of scalability, as it involves repeatedly acquiring preference data and training new reward models and LLMs for each individual's preferences. To address these challenges, we propose a new paradigm where users specify what they value most within the system message, steering the LLM's generation behavior to better align with the user's intentions. However, a naive application of such an approach is non-trivial since LLMs are typically trained on a uniform system message (e.g., "You are a helpful assistant") which limits their ability to generalize to diverse, unseen system messages. To improve this generalization, we create the Multifaceted Collection, a preference dataset with 192k combinations of values beyond generic helpfulness and harmlessness, spanning 65k user instructions. Using this dataset, we train a 7B LLM called Janus and test it on 921 prompts from 5 benchmarks (AlpacaEval 2.0, FLASK, Koala, MT-Bench, and Self-Instruct) by adding various unseen system messages that reflect user preferences. Janus achieves tie+win rate of 75.2%, 72.4%, and 66.4% against Mistral 7B Instruct v0.2, GPT-3.5 Turbo, and GPT-4, respectively. Unexpectedly, on three benchmarks focused on response helpfulness (AlpacaEval 2.0, MT-Bench, Arena Hard Auto v0.1), Janus also outperforms LLaMA 3 8B Instruct by a +4.0%, +0.1%, +3.0% margin, underscoring that training with a vast array of system messages could also enhance alignment to the general public's preference as well. Our code, dataset, benchmark, and models are available at https://github.com/kaistAI/Janus.

Motivation & Objective

• Address the scalability challenge in personalized LLM alignment, where retraining for each user’s preferences is computationally prohibitive.
• Overcome the limitation of open-source LLMs in generalizing to unseen system messages due to training on a single, uniform system prompt.
• Develop a method that enables LLMs to adapt to personalized preferences at inference time via system message prompting, eliminating the need for retraining.
• Demonstrate that training on diverse system messages improves not only personalization but also general alignment with public preferences.

Proposed method

• Construct the Multifaceted Collection, a preference dataset with 192k unique system message-instruction-response triplets, spanning 65k distinct instructions.
• Source instructions from five existing datasets and use GPT-4-Turbo to generate diverse system messages and reference responses reflecting multifaceted human preferences.
• Train a 7B LLM (Janus) on the Multifaceted Collection using instruction fine-tuning to enable generalization to unseen system messages during inference.
• Evaluate Janus across five benchmarks (AlpacaEval 2.0, FLASK, Koala, MT-Bench, Self-Instruct) by injecting diverse, unseen system messages into prompts.
• Use human-verified, high-quality system messages and responses, with low-quality instances filtered out prior to training and evaluation.
• Perform pairwise human evaluation and GPT-4-based automatic scoring to validate performance across multiple preference dimensions.

Figure 1: Previous LLMs are trained with homogeneous system messages reflecting general helpfulness and harmlessness. We propose training LLMs with diverse system messages, each representing an individual’s multifaceted preferences, to generalize to unseen system messages. The resulting model, Janus

Experimental results

Research questions

• RQ1 Can an LLM be trained to generalize to a wide range of unseen system messages that reflect individual user preferences without retraining?
• RQ2 Does training on diverse system messages improve not only personalization but also general alignment with broad public preferences?
• RQ3 To what extent can a single LLM, fine-tuned on varied system messages, outperform established models like Mistral 7B, GPT-3.5, and GPT-4 in preference alignment?
• RQ4 How diverse and non-overlapping are the preferences captured in the Multifaceted Collection across different dimensions and subdimensions?
• RQ5 Can system message generalization lead to improved performance on benchmarks focused on helpfulness, conciseness, and factual accuracy?

Key findings

• Janus achieves a tie+win rate of 75.2% against Mistral 7B Instruct v0.2, 72.4% against GPT-3.5 Turbo, and 66.4% against GPT-4 on benchmark evaluations with unseen system messages.
• In human pairwise evaluation, Janus wins 74.8% of comparisons against Mistral 7B Instruct v0.2, 70.8% against GPT-3.5-Turbo, and 57.9% against GPT-4-Turbo-0125.
• Janus outperforms LLaMA 3 8B Instruct by +4.0%, +0.1%, and +3.0% on AlpacaEval 2.0, MT-Bench, and Arena Hard Auto v0.1, respectively, demonstrating improved general alignment.
• ROUGE-L scores between preference pairs average 0.21 across all dimensions, with a maximum of 0.25, confirming significant diversity and low overlap in the 192k preference descriptions.
• The Multifaceted Collection captures 12 subdimensions across four preference dimensions—Style, Background Knowledge, Informativeness, and Harmlessness—each with distinct keywords and nuanced preferences.
• Janus achieves an average GPT-4 evaluation score of 4.24 out of 5.0, surpassing LLaMA 3 8B Instruct (4.1), Mixtral 8x7B Instruct (4.03), and GPT-3.5-Turbo-0125 (3.91), approaching LLaMA 3 70B Instruct (4.39).

Figure 2: Construction process of Multifaceted Collection . For each instruction, preference descriptions are augmented from general to specific, allowing for multiple facets to branch out. We combine preferences from various dimensions into a system message to materialize these preferences as model