[Paper Review] Democratic Preference Alignment via Sortition-Weighted RLHF

Suvadip Sana, Jinzhou Wu+1 more|arXiv (Cornell University)|Feb 4, 2026

Game Theory and Voting Systems1 citations

TL;DR

The paper introduces DemPO, a sortition-based framework for preference-based model fine-tuning, producing Hard Panel (sampled demographically representative panel) and Soft Panel (weights by inclusion probability) training to align AI values with a representative public. Hard Panel and Soft Panel outperform standard full-pool RLHF across model sizes and aggregation methods, with panel-based gains increasing with model capacity.

ABSTRACT

Whose values should AI systems learn? Preference based alignment methods like RLHF derive their training signal from human raters, yet these rater pools are typically convenience samples that systematically over represent some demographics and under represent others. We introduce Democratic Preference Optimization, or DemPO, a framework that applies algorithmic sortition, the same mechanism used to construct citizen assemblies, to preference based fine tuning. DemPO offers two training schemes. Hard Panel trains exclusively on preferences from a quota satisfying mini public sampled via sortition. Soft Panel retains all data but reweights each rater by their inclusion probability under the sortition lottery. We prove that Soft Panel weighting recovers the expected Hard Panel objective in closed form. Using a public preference dataset that pairs human judgments with rater demographics and a seventy five clause constitution independently elicited from a representative United States panel, we evaluate Llama models from one billion to eight billion parameters fine tuned under each scheme. Across six aggregation methods, the Hard Panel consistently ranks first and the Soft Panel consistently outperforms the unweighted baseline, with effect sizes growing as model capacity increases. These results demonstrate that enforcing demographic representativeness at the preference collection stage, rather than post hoc correction, yields models whose behavior better reflects values elicited from representative publics.

Motivation & Objective

• Address bias from convenience-sample rater pools in preference-based alignment.
• Introduce algorithmic sortition to construct demographically representative training signals.
• Propose Hard Panel and Soft Panel training schemes and relate their objectives.
• Evaluate representativeness-driven training using PRISM data and a representative U.S. constitution.
• Analyze model-size scaling of panel-based gains and provide diagnostics.

Proposed method

• Use LEXIMIN sortition to construct a lottery over quota-feasible panels that match demographic marginals.
• Define Hard Panel training on a single sampled panel S with per-rater normalization by N_i.
• Define Soft Panel weighting where each rater i is weighted by inclusion probability π_i from the sortition lottery.
• Relate Soft Panel objective to the expected Hard Panel objective with weights w_i.
• Train models with Direct Preference Optimization (DPO) on multi-turn PRISM data.
• Evaluate across six aggregation methods (Bradley–Terry, Plackett–Luce, Borda, Copeland, Kemeny-Young, Mallows) and with a 75-clause constitution.

Figure 1 : The DemPO pipeline for democratic preference alignment. A biased, self-selected pool of data labelers is transformed into a demographically representative mini-public via algorithmic sortition subject to population-derived quota constraints. Preferences from this representative panel (Har

Experimental results

Research questions

• RQ1 Does enforcing demographic representativeness at the preference-collection stage shift model behavior toward values elicited from representative publics?
• RQ2 How do Hard Panel and Soft Panel training compare to full PRISM and US-Rep baselines across model sizes?
• RQ3 Do representativeness-aware objectives align models with a constitution derived from representative public input?
• RQ4 How do panel-based gains scale with model size and aggregation method?

Key findings

• Hard Panel ranks highest across aggregation methods.
• Soft Panel consistently improves over the unweighted Full PRISM baseline.
• Hard Panel outperforms US-Rep, and gains grow with model size.
• Soft Panel gains over Full PRISM increase as model size grows (1B→3B→8B).
• Judge reliability shows substantial agreement across rankings (Kendall τ≈0.776, Fleiss’ κ≈0.710).
• Constitutional evaluation via automated judge indicates panel-based training aligns with representative public values.

Figure 2 : Model ranking under multiple aggregation methods (Llama-3.1-8B). Left: Borda and Copeland scores with 95% bootstrap confidence intervals, and Kemeny consensus summarized as rank-position probabilities under bootstrap resampling. Right: Bradley–Terry and Plackett–Luce log-ability scores wi