[论文解读] A Roadmap to Pluralistic Alignment

Taylor Sorensen, Jared Moore+10 more|arXiv (Cornell University)|Feb 7, 2024

Ethics and Social Impacts of AI被引用 10

一句话总结

本文定义 AI 模型的三种形式的多元化（Overton、Steerable、Distributional），提出三种相应的基准类别（multi-objective、trade-off steerable、jury-pluralistic），提出当前对齐可能降低 distributional pluralism 的实证担忧，并概述用于多元化评估与对齐的研究议程。

ABSTRACT

With increased power and prevalence of AI systems, it is ever more critical that AI systems are designed to serve all, i.e., people with diverse values and perspectives. However, aligning models to serve pluralistic human values remains an open research question. In this piece, we propose a roadmap to pluralistic alignment, specifically using language models as a test bed. We identify and formalize three possible ways to define and operationalize pluralism in AI systems: 1) Overton pluralistic models that present a spectrum of reasonable responses; 2) Steerably pluralistic models that can steer to reflect certain perspectives; and 3) Distributionally pluralistic models that are well-calibrated to a given population in distribution. We also formalize and discuss three possible classes of pluralistic benchmarks: 1) Multi-objective benchmarks, 2) Trade-off steerable benchmarks, which incentivize models to steer to arbitrary trade-offs, and 3) Jury-pluralistic benchmarks which explicitly model diverse human ratings. We use this framework to argue that current alignment techniques may be fundamentally limited for pluralistic AI; indeed, we highlight empirical evidence, both from our own experiments and from other work, that standard alignment procedures might reduce distributional pluralism in models, motivating the need for further research on pluralistic alignment.

研究动机与目标

• Motivate the importance of pluralism in AI alignment to serve diverse human values and perspectives.
• Formalize three operationalizations of pluralism in models: Overton, Steerable, and Distributional.
• Propose three classes of pluralistic benchmarks to evaluate models across diverse objectives and populations.
• Argue that current alignment techniques may reduce distributional pluralism and outline future research directions.

提出的方法

• Formal definitions of Overton pluralism (outputting the full set of reasonable answers) and mechanisms to operationalize it.
• Formal definitions of Steerable pluralism (conditioning responses on attributes or perspectives) and methods to measure faithfulness.
• Formal definitions of Distributional pluralism (matching a target population distribution over answers) and metrics to assess calibration.
• Definition of three benchmark families: multi-objective benchmarks, trade-off steerable benchmarks, and jury-pluralistic benchmarks.
• Discussion of alignment procedures and empirical observations suggesting that RLHF/post-alignment can reduce distributional pluralism.

Figure 1 : Three kinds of pluralism in models.

实验结果

研究问题

• RQ1 How can pluralism be defined and operationalized in AI systems beyond average human preference?
• RQ2 What benchmark designs are appropriate to measure pluralism in models (Overton, steerable, distributional)?
• RQ3 Do current alignment techniques (e.g., RLHF) reduce distributional pluralism, and under what conditions?
• RQ4 How can we implement and evaluate Overton, steerable, and distributional pluralism in practical LLM applications?
• RQ5 What future research is needed to develop pluralistic evaluations and alignment strategies?

主要发现

• Three formalizations of pluralism for models: Overton (whole spectrum of reasonable answers), Steerable (attribute-faithful steering), Distributional (population-calibrated distributions).
• Three benchmark classes proposed: multi-objective benchmarks, trade-off steerable benchmarks, and jury-pluralistic benchmarks for explicit modeling of diverse ratings.
• Empirical and theoretical indications that standard alignment may reduce distributional pluralism, motivating further research into pluralistic evaluation and alignment approaches.
• Discussion of practical limitations and applications for each pluralism type and benchmark class.
• A roadmap and recommendations for future work toward pluralistic evaluations and alignment.

Figure 2 : Three kinds of pluralistic benchmarks.