[Paper Review] A Roadmap to Pluralistic Alignment

Taylor Sorensen, Jared Moore+10 more|arXiv (Cornell University)|Feb 7, 2024

Ethics and Social Impacts of AI10 citations

TL;DR

The paper defines three forms of pluralism for AI models (Overton, Steerable, Distributional), proposes three corresponding benchmark classes (multi-objective, trade-off steerable, jury-pluralistic), presents empirical concerns that current alignment may reduce distributional pluralism, and outlines a research agenda for pluralistic evaluation and alignment.

ABSTRACT

With increased power and prevalence of AI systems, it is ever more critical that AI systems are designed to serve all, i.e., people with diverse values and perspectives. However, aligning models to serve pluralistic human values remains an open research question. In this piece, we propose a roadmap to pluralistic alignment, specifically using language models as a test bed. We identify and formalize three possible ways to define and operationalize pluralism in AI systems: 1) Overton pluralistic models that present a spectrum of reasonable responses; 2) Steerably pluralistic models that can steer to reflect certain perspectives; and 3) Distributionally pluralistic models that are well-calibrated to a given population in distribution. We also formalize and discuss three possible classes of pluralistic benchmarks: 1) Multi-objective benchmarks, 2) Trade-off steerable benchmarks, which incentivize models to steer to arbitrary trade-offs, and 3) Jury-pluralistic benchmarks which explicitly model diverse human ratings. We use this framework to argue that current alignment techniques may be fundamentally limited for pluralistic AI; indeed, we highlight empirical evidence, both from our own experiments and from other work, that standard alignment procedures might reduce distributional pluralism in models, motivating the need for further research on pluralistic alignment.

Motivation & Objective

• Motivate the importance of pluralism in AI alignment to serve diverse human values and perspectives.
• Formalize three operationalizations of pluralism in models: Overton, Steerable, and Distributional.
• Propose three classes of pluralistic benchmarks to evaluate models across diverse objectives and populations.
• Argue that current alignment techniques may reduce distributional pluralism and outline future research directions.

Proposed method

• Formal definitions of Overton pluralism (outputting the full set of reasonable answers) and mechanisms to operationalize it.
• Formal definitions of Steerable pluralism (conditioning responses on attributes or perspectives) and methods to measure faithfulness.
• Formal definitions of Distributional pluralism (matching a target population distribution over answers) and metrics to assess calibration.
• Definition of three benchmark families: multi-objective benchmarks, trade-off steerable benchmarks, and jury-pluralistic benchmarks.
• Discussion of alignment procedures and empirical observations suggesting that RLHF/post-alignment can reduce distributional pluralism.

Figure 1 : Three kinds of pluralism in models.

Experimental results

Research questions

• RQ1 How can pluralism be defined and operationalized in AI systems beyond average human preference?
• RQ2 What benchmark designs are appropriate to measure pluralism in models (Overton, steerable, distributional)?
• RQ3 Do current alignment techniques (e.g., RLHF) reduce distributional pluralism, and under what conditions?
• RQ4 How can we implement and evaluate Overton, steerable, and distributional pluralism in practical LLM applications?
• RQ5 What future research is needed to develop pluralistic evaluations and alignment strategies?

Key findings

• Three formalizations of pluralism for models: Overton (whole spectrum of reasonable answers), Steerable (attribute-faithful steering), Distributional (population-calibrated distributions).
• Three benchmark classes proposed: multi-objective benchmarks, trade-off steerable benchmarks, and jury-pluralistic benchmarks for explicit modeling of diverse ratings.
• Empirical and theoretical indications that standard alignment may reduce distributional pluralism, motivating further research into pluralistic evaluation and alignment approaches.
• Discussion of practical limitations and applications for each pluralism type and benchmark class.
• A roadmap and recommendations for future work toward pluralistic evaluations and alignment.

Figure 2 : Three kinds of pluralistic benchmarks.