[Paper Review] Fine-tuning language models to find agreement among humans with diverse preferences

Michiel A. Bakker, Martin J. Chadwick+9 more|arXiv (Cornell University)|Nov 28, 2022

Topic ModelingComputer Science109 citations

TL;DR

The authors fine-tune a 70B language model to generate consensus statements that maximize group approval among diverse opinions, achieving higher approval than baselines and human opinions (>70% vs baselines, >65% vs best human).

ABSTRACT

Recent work in large language modeling (LLMs) has used fine-tuning to align outputs with the preferences of a prototypical user. This work assumes that human preferences are static and homogeneous across individuals, so that aligning to a a single "generic" user will confer more general alignment. Here, we embrace the heterogeneity of human preferences to consider a different challenge: how might a machine help people with diverse views find agreement? We fine-tune a 70 billion parameter LLM to generate statements that maximize the expected approval for a group of people with potentially diverse opinions. Human participants provide written opinions on thousands of questions touching on moral and political issues (e.g., "should we raise taxes on the rich?"), and rate the LLM's generated candidate consensus statements for agreement and quality. A reward model is then trained to predict individual preferences, enabling it to quantify and rank consensus statements in terms of their appeal to the overall group, defined according to different aggregation (social welfare) functions. The model produces consensus statements that are preferred by human users over those from prompted LLMs (>70%) and significantly outperforms a tight fine-tuned baseline that lacks the final ranking step. Further, our best model's consensus statements are preferred over the best human-generated opinions (>65%). We find that when we silently constructed consensus statements from only a subset of group members, those who were excluded were more likely to dissent, revealing the sensitivity of the consensus to individual contributions. These results highlight the potential to use LLMs to help groups of humans align their values with one another.

Motivation & Objective

• Investigate whether LLMs can help groups with diverse preferences find agreement on policy questions.
• Collect a large, diverse set of human opinions and use them to train models via reinforcement-like reranking.
• Develop a reward model to predict individual agreement with consensus statements.
• Explore the use of social welfare functions to aggregate individual preferences into a group consensus.
• Evaluate generalization to out-of-distribution questions and sensitivity to excluded opinions.

Proposed method

• Generate candidate consensus statements from group opinions using a 70B pretrained LLM (Chinchilla) in a prompt-driven loop.
• Create a supervised fine-tuned model (SFT) on high-quality consensus candidates to stabilize generation.
• Train a reward model to predict individual agreement with a given consensus statement, conditional on the person’s opinion.
• Rerank multiple candidate statements by predicted welfare under a chosen social welfare function to select the best one.
• Sample the inequality aversion parameter alpha during training to cover Utilitarian–Rawlsian spectrum.
• Evaluate consensus statements via human ratings of agreement and quality, and compare against baselines and human opinions.

Experimental results

Research questions

• RQ1 Can a language model generate consensus statements that are preferred by groups with diverse opinions?
• RQ2 Does optimizing a social welfare function (e.g., Utilitarian, Rawlsian) influence the quality and divisiveness of consensus statements?
• RQ3 Do models generalize to out-of-distribution questions not seen during training?
• RQ4 How sensitive is the consensus to the specific opinions included in the prompt (opinion exclusion effect)?

Key findings

• The SFT-Utilitarian model outperforms baselines on both mean group agreement and minimum (worst-off) agreement ratings.
• The model’s consensus statements are preferred over human-generated opinions in aggregate ratings (mean up to >65% win rate).
• Across within- and out-of-distribution questions, the SFT-Utilitarian model maintains strong performance, indicating generalization capability.
• Excluding some participants’ opinions during consensus generation reduces projected group agreement by an average of 0.47 Likert points (on the inclusion vs. exclusion comparison).
• Quality ratings improve across the pipeline (SFT and reward modeling increase perceived quality).
• In approximately 50% of rounds, position statements were non-divisive, yet the model produced consensus statements that reduced divisiveness in many divisive rounds (e.g., 65.6% less divisive than initial stance).