[Paper Review] Large Language Models Can Be Used to Estimate the Latent Positions of Politicians

Patrick Y. Wu, Jonathan Nagler+2 more|arXiv (Cornell University)|Mar 21, 2023

Computational and Text Analysis Methods29 citations

TL;DR

The paper introduces LaMP, a pairwise comparison method using instruction-tuned LLMs to estimate latent politicians’ positions on ideology, gun control, and abortion, scaled with the Bradley-Terry model, and validates these scales against existing measures and predictive voting data.

ABSTRACT

Existing approaches to estimating politicians' latent positions along specific dimensions often fail when relevant data is limited. We leverage the embedded knowledge in generative large language models (LLMs) to address this challenge and measure lawmakers' positions along specific political or policy dimensions. We prompt an instruction/dialogue-tuned LLM to pairwise compare lawmakers and then scale the resulting graph using the Bradley-Terry model. We estimate novel measures of U.S. senators' positions on liberal-conservative ideology, gun control, and abortion. Our liberal-conservative scale, used to validate LLM-driven scaling, strongly correlates with existing measures and offsets interpretive gaps, suggesting LLMs synthesize relevant data from internet and digitized media rather than memorizing existing measures. Our gun control and abortion measures -- the first of their kind -- differ from the liberal-conservative scale in face-valid ways and predict interest group ratings and legislator votes better than ideology alone. Our findings suggest LLMs hold promise for solving complex social science measurement problems.

Motivation & Objective

• Motivate a novel approach to measuring latent legislative positions when data are sparse or incomplete.
• Leverage embedded knowledge in instruction-tuned LLMs to perform pairwise comparisons of senators.
• Scale LLM responses into continuous latent position scores using the Bradley-Terry model.
• Validate the resulting LaMP scores against established ideology measures and external outcomes.
• Demonstrate the potential of LLM-based scaling for issue-specific political dimensions.

Proposed method

• Create pairwise matchups between all senators in the 116th Congress for each target dimension.
• Prompt an instruction/dialogue-tuned LLM (ChatGPT-3.5) to select a “winner” in each matchup for the given dimension (liberal-conservative ideology, gun control, abortion rights).
• Apply the Bradley-Terry model to convert pairwise winners/losers into a unidimensional latent scale (LaMP scores).
• Use a zero-shot setting with names, party, and state as the only inputs to the LLM for each matchup, without providing examples.
• Assess stability by repeating matchups across three iterations and computing inter-iteration correlations.
• Validate Ideology LaMP against DW-NOMINATE and perceived ideology/CFscores, then validate Gun Control LaMP and Abortion Rights LaMP against NRA/NARAL outcomes and relevant votes.

Experimental results

Research questions

• RQ1 Can a generative LLM produce consistent pairwise judgments about lawmakers on predefined political dimensions?
• RQ2 Do LLM-derived scales align with established ideological measures like DW-NOMINATE and perceived ideology scores?
• RQ3 Can issue-specific LaMP scales (gun control, abortion rights) provide predictive validity for external outcomes (NRA grades, NARAL grades, and related votes)?
• RQ4 Do LaMP scores capture information beyond traditional behavior-based or perception-based ideology measures?

Key findings

• Ideology LaMP scores correlate highly with the first dimension of DW-NOMINATE (0.967 overall).
• Ideology LaMP scores correlate with perceived ideology (0.941) and CFscores (0.933).
• Gun Control LaMP scores predict NRA grades and the 2022 Bipartisan Safer Communities Act votes, showing predictive validity beyond DW-NOMINATE (e.g., 0.214 when predicting NRA votes for Republicans using Gun Control LaMP alone).
• Abortion Rights LaMP scores align with known pro-choice/ pro-life positions and predict NARAL grades better than DW-NOMINATE.
• LaMP scores are stable across iterations (Ideology LaMP correlations: ≥0.997; Gun Control LaMP correlations: ≥0.993; Abortion Rights LaMP correlations: ≥0.996).
• LaMP scales reveal differences from pure party-line ideology and can capture nuances such as conservative voting for ideological reasons or issue-specific positions (e.g., Manchin vs. Collins on abortion).