[Paper Review] Identification, Interpretability, and Bayesian Word Embeddings

Adam M. Lauretig|arXiv (Cornell University)|Apr 2, 2019

Topic Modeling41 references4 citations

TL;DR

This paper introduces Bayesian Word Embeddings with Automatic Relevance Determination to address identification and interpretability issues in standard word embeddings for social science research. By modeling embeddings as Bayesian latent variables and anchoring dimensions using theory-driven words, the method enables regression-ready, interpretable word representations, which reveal a post-1945 decline in internationalist rhetoric in U.S. inaugural addresses and a significant link between elite bellicosity in diplomatic documents and increased U.S. hostile foreign policy actions.

ABSTRACT

Social scientists have recently turned to analyzing text using tools from natural language processing like word embeddings to measure concepts like ideology, bias, and affinity. However, word embeddings are difficult to use in the regression framework familiar to social scientists: embeddings are are neither identified, nor directly interpretable. I offer two advances on standard embedding models to remedy these problems. First, I develop Bayesian Word Embeddings with Automatic Relevance Determination priors, relaxing the assumption that all embedding dimensions have equal weight. Second, I apply work identifying latent variable models to anchor the dimensions of the resulting embeddings, identifying them, and making them interpretable and usable in a regression. I then apply this model and anchoring approach to two cases, the shift in internationalist rhetoric in the American presidents' inaugural addresses, and the relationship between bellicosity in American foreign policy decision-makers' deliberations. I find that inaugural addresses became less internationalist after 1945, which goes against the conventional wisdom, and that an increase in bellicosity is associated with an increase in hostile actions by the United States, showing that elite deliberations are not cheap talk, and helping confirm the validity of the model.

Motivation & Objective

• To resolve the lack of identification and interpretability in standard word embeddings, which hinders their use in regression models common in social science research.
• To develop a Bayesian framework for word embeddings that allows for dimension-specific regularization via Automatic Relevance Determination priors.
• To anchor embedding dimensions using theory-driven words, enabling identification and interpretability for causal inference.
• To apply the method to two real-world corpora: U.S. presidential inaugural addresses and declassified Foreign Relations of the United States (FRUS) diplomatic documents.
• To validate the model by showing that its measures correlate with external conflict data and reveal novel historical trends.

Proposed method

• Formalizes word embeddings as a Bayesian latent variable model, using variational Bayesian inference for posterior estimation.
• Incorporates Automatic Relevance Determination (ARD) priors on embedding dimensions to allow differential weighting and identify irrelevant dimensions.
• Applies identification techniques from ideal-point modeling (e.g., Rivers, 2003; Clinton et al., 2004) to anchor embedding dimensions using semantically meaningful words.
• Uses a two-step anchoring process: first, identify high-contrast word pairs (e.g., 'peace' vs. 'war') to define dimension endpoints; second, scale all words along these dimensions.
• Employs a Poisson generalized linear model to test the causal relationship between embedded bellicosity and material conflict events in diplomatic documents.
• Validates the model by comparing its output to external conflict event counts and assessing model fit and robustness.

Experimental results

Research questions

• RQ1 Has the rhetorical emphasis on internationalism in U.S. presidential inaugural addresses declined since 1945, and does this shift contradict conventional wisdom in international relations?
• RQ2 To what extent does the level of bellicosity in elite diplomatic deliberations predict actual U.S. foreign policy hostility?
• RQ3 Can Bayesian word embeddings with ARD priors and dimension anchoring be used to create interpretable, regression-ready measures of semantic concepts in political text?
• RQ4 Do the embedded measures of rhetoric correlate with independent datasets on conflict behavior, thereby validating the model’s construct validity?
• RQ5 How can word embeddings be made identifiable and interpretable for use in causal social science inference?

Key findings

• The analysis of inaugural addresses reveals a statistically significant decline in internationalist rhetoric after 1945, contradicting conventional expectations of growing global engagement.
• The Bayesian Word Embedding model successfully identifies and interprets embedding dimensions using theory-driven word anchors, enabling regression use.
• A one-standard-deviation increase in the previous bi-weekly bellicosity score is associated with a statistically significant increase in U.S.-initiated hostile events, as shown by a Poisson GLM with 95% confidence intervals.
• The model’s bellicosity scale correlates with external conflict data, supporting its validity and reliability as a measure of elite rhetoric.
• The method enables detection of semantic shifts—such as the decline in internationalism—that are undetectable with traditional document-level text-as-data methods.
• The use of ARD priors reduces noise by identifying and downweighting irrelevant embedding dimensions, improving model interpretability and efficiency.