[Paper Review] Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias

Jesse Vig, Sebastian Gehrmann+7 more|arXiv (Cornell University)|Apr 26, 2020

Topic ModelingComputer Science75 references66 citations

TL;DR

This paper introduces causal mediation analysis to interpret neural NLP models by treating internal components (neurons, attention heads) as mediators between input and output, and applies it to gender bias in Transformer-based LMs.

ABSTRACT

Common methods for interpreting neural models in natural language processing typically examine either their structure or their behavior, but not both. We propose a methodology grounded in the theory of causal mediation analysis for interpreting which parts of a model are causally implicated in its behavior. It enables us to analyze the mechanisms by which information flows from input to output through various model components, known as mediators. We apply this methodology to analyze gender bias in pre-trained Transformer language models. We study the role of individual neurons and attention heads in mediating gender bias across three datasets designed to gauge a model's sensitivity to gender bias. Our mediation analysis reveals that gender bias effects are (i) sparse, concentrated in a small part of the network; (ii) synergistic, amplified or repressed by different components; and (iii) decomposable into effects flowing directly from the input and indirectly through the mediators.

Motivation & Objective

• Motivate and formalize causal mediation analysis for interpreting neural NLP models.
• Investigate how internal components (neurons, attention heads) causally contribute to gender bias in pre-trained Transformers.
• Provide empirical evidence on sparsity, synergism, and decomposability of bias effects within models.

Proposed method

• Model the neural network as a DAG with input as root and output as leaf.
• Define do-interventions on inputs (set-gender) and mediators (specific neurons or attention heads).
• Compute total effects (TE), natural direct effects (NDE), and natural indirect effects (NIE) to quantify causal roles of mediators.
• Apply interventions at neuron and attention-head levels to measure their contributions to bias across GPT2 variants and other models.
• Use datasets including Professions, Winobias, and Winogender to assess grammatical gender bias.

Experimental results

Research questions

• RQ1 How is gender bias in language models mediated by internal components such as neurons and attention heads?
• RQ2 Are bias effects sparse, synergistic, and decomposable into direct and indirect pathways within Transformer models?
• RQ3 Do findings generalize across model sizes and architectures beyond GPT2?
• RQ4 How do different bias measurement datasets influence observed mediation effects?

Key findings

• Bias effects are larger in bigger models and can saturate in the largest GPT2 variants.
• Gender bias is concentrated in a small subset of model components, indicating sparsity.
• Mediator interactions can amplify or dampen bias, showing synergistic effects among components.
• The total effect can be well-approximated by summing direct and indirect effects, indicating decomposability despite nonlinearity.
• Findings generalize across autoregressive models and, to a lesser extent, masked LMs.