[Paper Review] Identifying and Controlling Important Neurons in Neural Machine Translation

Anthony Bau, Yonatan Belinkov+4 more|arXiv (Cornell University)|Nov 3, 2018

Natural Language Processing Techniques110 citations

TL;DR

The paper develops unsupervised methods to identify and analyze important neurons in NMT models across multiple language pairs, showing these neurons carry interpretable linguistic information and enabling partial control of translations by neuron activations.

ABSTRACT

Neural machine translation (NMT) models learn representations containing substantial linguistic information. However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons. We develop unsupervised methods for discovering important neurons in NMT models. Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision. We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena. Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.

Motivation & Objective

• Determine whether individual neurons in NMT models carry important and interpretable linguistic information.
• Develop unsupervised methods to rank neurons by their contribution to translation quality without external supervision.
• Investigate whether activating or erasing selected neurons can predictably influence translation outcomes.

Proposed method

• Train multiple English→(5 languages) NMT models with charCNN input on UN corpus.
• Rank neurons using four unsupervised methods: MaxCorr, MinCorr, LinReg, and SVCCA, across model pairs.
• Verify neuron importance via erasure experiments and BLEU evaluation.
• Analyze linguistic properties captured by top neurons through visualization and predictive tasks.
• Demonstrate controlling translations by modifying activations of selected neurons and evaluating success rates and BLEU impact.

Experimental results

Research questions

• RQ1 Are there individual neurons in NMT models that significantly impact translation quality when erased?
• RQ2 Do highly ranked neurons encode interpretable linguistic information such as tense, number, or gender?
• RQ3 Can translations be steered predictably by modifying activations of specific neurons?

Key findings

• Erasing top-ranked neurons significantly degrades translation quality, with BLEU drops of 15–20 points when removing the top 10% vs 2–3 points for the bottom 10% (for MaxCorr, MinCorr, LinReg).
• Top SVCCA directions cause rapid degradation (about 15 BLEU points for 5 directions, 1% of directions) and tend to align with specific words.
• MaxCorr tends to reveal position-related information, while LinReg and SVCCA capture word identity properties; many top neurons encode tense, numbers, or parentheses-related tokens.
• You can influence translations by modifying activations of selected neurons, achieving up to 67% success in changing tense (past to present) with modest BLEU loss (~2 points), but gender control remains harder (~21% success).
• Neurons often correspond to linguistically meaningful phenomena across models, supporting the idea that some information in NMT is localized to interpretable units rather than fully distributed.