[Paper Review] Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks

Yonatan Belinkov, Lluı́s Màrquez+4 more|arXiv (Cornell University)|Jan 23, 2018

Natural Language Processing TechniquesComputer Science26 references81 citations

TL;DR

The paper probes how NMT encoder layers encode POS and semantic tagging information by using layer-wise NMT representations as features for word-level POS and SEM tagging classifiers, revealing that lower layers favor POS while higher layers capture semantic information.

ABSTRACT

While neural machine translation (NMT) models provide improved translation quality in an elegant, end-to-end framework, it is less clear what they learn about language. Recent work has started evaluating the quality of vector representations learned by NMT models on morphological and syntactic tasks. In this paper, we investigate the representations learned at different layers of NMT encoders. We train NMT systems on parallel data and use the trained models to extract features for training a classifier on two tasks: part-of-speech and semantic tagging. We then measure the performance of the classifier as a proxy to the quality of the original NMT model for the given task. Our quantitative analysis yields interesting insights regarding representation learning in NMT models. For instance, we find that higher layers are better at learning semantics while lower layers tend to be better for part-of-speech tagging. We also observe little effect of the target language on source-side representations, especially with higher quality NMT models.

Motivation & Objective

• Determine which encoder layers of NMT models learn useful representations for POS tagging.
• Determine which encoder layers of NMT models learn useful representations for semantic tagging (SEM).
• Assess how target language and model architecture influence source-side representations in NMT.
• Compare NMT-derived representations to baselines and an encoder-decoder Word2Tag upper bound.

Proposed method

• Train NMT systems on fully-aligned UN parallel data across six target languages.
• Extract encoder layer representations for each source word from pre-trained NMT models.
• Train a word-level classifier using layer-specific features to predict POS or SEM tags.
• Evaluate classifier accuracy as a proxy for the quality of NMT representations for each task.
• Compare across layers, target languages, and architectures (uni/bidirectional, residuals).
• Use baselines (most frequent tag, unsupervised embeddings) and an encoder-decoder Word2Tag upper bound for context.

Experimental results

Research questions

• RQ1 Which encoder layers in NMT models yield the best representations for POS tagging?
• RQ2 Which encoder layers yield the best representations for SEM tagging?
• RQ3 How does the target language affect source-side representations when training NMT models?
• RQ4 Do architectural variants (bidirectional encoders, residual connections) alter the layer-wise representation quality for POS and SEM tagging?
• RQ5 Is there a difference in representation quality between English autoencoders and MT-trained encoders for these tasks?

Key findings

• Lower encoder layers are usually better for POS tagging.
• Higher encoder layers better capture semantic information for SEM tagging.
• 4th-layer representations improve SEM tagging across target languages (except Chinese) and often outperform lower layers.
• Target language has a small influence on source representations when data is large; the effect is more noticeable with smaller training data.
• Bidirectional encoders and residual connections generally improve both translation quality and SEM tagging performance, with SEM benefiting more from higher layers.