[Paper Review] Understanding the Political Ideology of Legislators from Social Media Images

Nan Miles Xi, Di Ma+4 more|arXiv (Cornell University)|Jul 22, 2019

Computational and Text Analysis Methods12 citations

TL;DR

This study uses deep learning to classify U.S. legislators' political ideology—Republican or Democratic—exclusively from their Facebook profile images, achieving 82.35% accuracy by aggregating multiple photos per individual. The model's predictions correlate strongly with traditional ideological measures, revealing that conservative images emphasize institutional authority, dominant social groups, and higher happiness, while liberal images feature diversity and progressive symbols.

ABSTRACT

In this paper, we seek to understand how politicians use images to express ideological rhetoric through Facebook images posted by members of the U.S. House and Senate. In the era of social media, politics has become saturated with imagery, a potent and emotionally salient form of political rhetoric which has been used by politicians and political organizations to influence public sentiment and voting behavior for well over a century. To date, however, little is known about how images are used as political rhetoric. Using deep learning techniques to automatically predict Republican or Democratic party affiliation solely from the Facebook photographs of the members of the 114th U.S. Congress, we demonstrate that predicted class probabilities from our model function as an accurate proxy of the political ideology of images along a left-right (liberal-conservative) dimension. After controlling for the gender and race of politicians, our method achieves an accuracy of 59.28% from single photographs and 82.35% when aggregating scores from multiple photographs (up to 150) of the same person. To better understand image content distinguishing liberal from conservative images, we also perform in-depth content analyses of the photographs. Our findings suggest that conservatives tend to use more images supporting status quo political institutions and hierarchy maintenance, featuring individuals from dominant social groups, and displaying greater happiness than liberals.

Motivation & Objective

• To investigate whether political ideology can be inferred from social media images of legislators, particularly through visual cues in Facebook profile photos.
• To assess the extent to which visual features in images—such as clothing, symbols, and facial expressions—correlate with liberal or conservative political orientation.
• To compare human and machine classification accuracy in identifying party affiliation from images, and to examine differences in feature interpretation.
• To identify and analyze the most predictive visual elements distinguishing liberal from conservative images in political communication.

Proposed method

• Employed a convolutional neural network (CNN) to classify the political party affiliation (Republican or Democratic) of members of the 114th U.S. Congress based solely on their Facebook profile images.
• Trained the model on a dataset of 272 legislators, using single images and aggregated scores from up to 150 images per individual to improve prediction reliability.
• Validated model predictions against DW-NOMINATE scores—the gold standard for measuring political ideology—demonstrating strong correlation between image-based predictions and established ideological rankings.
• Conducted in-depth content analysis of high-scoring images to identify salient visual features such as military uniforms, clothing, symbols (e.g., Capitol Building), and demographic representation.
• Used human judges to label images and compare their classifications with model outputs, identifying cases of human bias (e.g., racial stereotyping) and model limitations (e.g., misreading text or rare symbols).

Experimental results

Research questions

• RQ1 Can machine learning accurately predict the political party affiliation of legislators using only their Facebook profile images?
• RQ2 How does the performance of a deep learning classifier compare to human judgment in identifying party affiliation from images?
• RQ3 Which specific visual features—such as clothing, symbols, or facial expressions—are most predictive of liberal or conservative political ideology in political images?
• RQ4 To what extent do image-based predictions of ideology correlate with established ideological measures like DW-NOMINATE scores?

Key findings

• The deep learning model achieved 59.28% accuracy in classifying party affiliation from a single photograph and 82.35% accuracy when aggregating predictions across multiple images of the same legislator.
• Model predictions showed a strong correlation with DW-NOMINATE scores, validating the model’s output as a reliable proxy for left–right political ideology.
• Conservative images were more likely to feature symbols of institutional authority (e.g., the Capitol Building), military personnel, and individuals from dominant racial and gender groups, with higher expressions of happiness.
• Liberal images were more likely to include diverse racial and ethnic groups, symbols of inclusion (e.g., rainbow flags), and depictions of children or community service.
• Humans outperformed the model in interpreting rare or context-specific symbols (e.g., Habitat for Humanity t-shirts), but also exhibited racial and gender-based stereotyping in their judgments.
• The model failed to recognize textual elements in images (e.g., 'DEFUND OBAMACARE'), while humans sometimes correctly interpreted such content despite low visual frequency in the training data.