[Paper Review] Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis

Kelly Zhang, Samuel R. Bowman|arXiv (Cornell University)|Sep 26, 2018

Topic Modeling22 references41 citations

TL;DR

The paper compares four pretraining objectives (language modeling, translation, skip-thought, autoencoding) and shows bidirectional language models yield the strongest syntactic representations for POS tagging and CCG supertagging, often outperforming translation encoders and even rivaling random LSTMs when data is abundant for auxiliary tasks.

ABSTRACT

Recent work using auxiliary prediction task classifiers to investigate the properties of LSTM representations has begun to shed light on why pretrained representations, like ELMo (Peters et al., 2018) and CoVe (McCann et al., 2017), are so beneficial for neural language understanding models. We still, though, do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn. With this in mind, we compare four objectives---language modeling, translation, skip-thought, and autoencoding---on their ability to induce syntactic and part-of-speech information. We make a fair comparison between the tasks by holding constant the quantity and genre of the training data, as well as the LSTM architecture. We find that representations from language models consistently perform best on our syntactic auxiliary prediction tasks, even when trained on relatively small amounts of data. These results suggest that language modeling may be the best data-rich pretraining task for transfer learning applications requiring syntactic information. We also find that the representations from randomly-initialized, frozen LSTMs perform strikingly well on our syntactic auxiliary tasks, but this effect disappears when the amount of training data for the auxiliary tasks is reduced.

Motivation & Objective

• Motivate understanding of how pretraining objectives shape learned linguistic representations.
• Fairly compare pretraining tasks by controlling data source, volume, and model architecture.
• Evaluate syntactic knowledge in pretrained representations using auxiliary classifiers for POS tagging and CCG supertagging.
• Examine the effect of training data volume and randomness on learned representations.

Proposed method

• Train multiple LSTM-based models on English-German translation data and monolingual data across four objectives: language modeling (LM), translation, skip-thought, and autoencoding.
• Use bidirectional LM (BiLM) representations by concatenating forward and backward LM hidden states as token representations.
• Freeze pretrained encoders and train auxiliary classifiers (MLPs) on POS tagging and CCG supertagging to probe syntactic information in hidden states.
• Compare against untrained (randomly initialized) LSTMs and a WC-MFC baseline to isolate learned vs memorized information.
• Vary training data volume (1M, 5M, 15M, 63M sentences) and classifier data (1%, 10%, 100%) to study data effects.
• Control for data domain by using the WSJ/PTB and CCG Bank datasets for POS and CCG tagging.

Experimental results

Research questions

• RQ1 How does the training task (LM, translation, skip-thought, autoencoding) affect the encoding of syntactic information?
• RQ2 Does the amount of training data influence the ability of pretrained representations to support syntactic auxiliary tasks?
• RQ3 Are randomly initialized encoders capable of supporting syntactic tagging when auxiliary classifiers are given abundant data?
• RQ4 How do layer and architecture choices affect which syntactic information is captured in hidden representations?
• RQ5 Do bidirectional contexts (BiLMs) provide advantages over unidirectional or translation-based encoders for syntactic transfer?

Key findings

• Bidirectional language models (BiLMs) consistently outperform other tasks (translation, skip-thought, autoencoding) on POS tagging and CCG supertagging across data regimes.
• BiLMs trained on as little as 1 million sentences can outperform translation and skip-thought models trained on much larger data, indicating data-efficient syntax learning.
• Even with the same data, BiLMs often beat translation encoders, and the advantage is larger for CCG supertagging than POS tagging.
• Randomly initialized LSTMs perform surprisingly well when the auxiliary classifier has abundant data, but their performance collapses with limited classifier data, indicating memorization rather than genuine syntactic encoding.
• Word identity probing reveals trained encoders outperform untrained ones on tagging tasks, confirming that learned representations capture more than simple neighbor word information.
• Lower LSTM layers store more immediate neighbor information, while higher layers encode more distant context, suggesting depth expands the receptive field for syntactic structure.