[Paper Review] Language Models Trained on Media Diets Can Predict Public Opinion

Eric Chu, Jacob Andreas+2 more|arXiv (Cornell University)|Mar 28, 2023

Computational and Text Analysis Methods23 citations

TL;DR

Language models adapted to specific media diets can predict subpopulation survey responses on COVID-19 and consumer confidence, with correlations around r=0.46 and robustness across prompts and media sources.

ABSTRACT

Public opinion reflects and shapes societal behavior, but the traditional survey-based tools to measure it are limited. We introduce a novel approach to probe media diet models -- language models adapted to online news, TV broadcast, or radio show content -- that can emulate the opinions of subpopulations that have consumed a set of media. To validate this method, we use as ground truth the opinions expressed in U.S. nationally representative surveys on COVID-19 and consumer confidence. Our studies indicate that this approach is (1) predictive of human judgements found in survey response distributions and robust to phrasing and channels of media exposure, (2) more accurate at modeling people who follow media more closely, and (3) aligned with literature on which types of opinions are affected by media consumption. Probing language models provides a powerful new method for investigating media effects, has practical applications in supplementing polls and forecasting public opinion, and suggests a need for further study of the surprising fidelity with which neural language models can predict human responses.

Motivation & Objective

• Motivate a new method to predict public opinion using language models trained on subpopulation media diets.
• Demonstrate that fine-tuned models on media content can predict survey response distributions for COVID-19 and consumer confidence.
• Assess robustness to question paraphrasing and to different media sources and attention levels.

Proposed method

• Fine-tune a base language model (e.g., BERT) on a media diet dataset from specific sources (online news, TV, radio).
• Probe the media-diet models with fill-in-the-blank prompts derived from survey questions and compute target-word probabilities.
• Normalize and group probabilities over synonyms to derive media-diet scores.
• Fit regressions to map media-diet scores (and optionally attention to news) to actual survey response proportions.
• Trace model predictions to training data via nearest-neighbor analysis in embedding space.

Experimental results

Research questions

• RQ1 RQ1a: Do media-diet models have predictive power for survey responses?
• RQ2 RQ1b: Are pretrained neural models necessary, or are simpler models sufficient?
• RQ3 RQ1c: Is synonym-grouping necessary for scoring media diets?
• RQ4 RQ1d: Are results robust to paraphrasing of prompts?
• RQ5 RQ2: Do media-diet models vary with level of attention to news or across media sources?
• RQ6 RQ3: Are certain topics or question types more strongly predicted by media-diet models?

Key findings

• Media-diet scores correlate with survey proportions (r = 0.458, 95% CI [0.350, 0.553]).
• A regression using media-diet scores significantly predicts survey proportions (beta = 0.115, 95% CI [0.087, 0.142]).
• Synonym-grouping improves correlations (r = 0.458 vs. r = 0.190 without grouping).
• Models adapted to media content outperform baseline BERT (r = 0.274 for BERT alone; r = 0.458 for Media Diet BERT).
• Combining media-diet scores with attention to news yields higher predictive power (beta = 0.523, CI [0.164, 0.882]; R2 = 0.3327).
• Predictive power holds across online, TV, and radio sources, and across different prompts and paraphrase methods.