[Paper Review] Constituency Parsing with a Self-Attentive Encoder

Nikita Kitaev, Dan Klein|arXiv (Cornell University)|May 2, 2018

Natural Language Processing Techniques9 references44 citations

TL;DR

The paper replaces an LSTM encoder with a self-attentive encoder in a discriminative constituency parser, achieving state-of-the-art results on WSJ and strong multilingual performance, and analyzes how separating content and position attention improves parsing.

ABSTRACT

We demonstrate that replacing an LSTM encoder with a self-attentive architecture can lead to improvements to a state-of-the-art discriminative constituency parser. The use of attention makes explicit the manner in which information is propagated between different locations in the sentence, which we use to both analyze our model and propose potential improvements. For example, we find that separating positional and content information in the encoder can lead to improved parsing accuracy. Additionally, we evaluate different approaches for lexical representation. Our parser achieves new state-of-the-art results for single models trained on the Penn Treebank: 93.55 F1 without the use of any external data, and 95.13 F1 when using pre-trained word representations. Our parser also outperforms the previous best-published accuracy figures on 8 of the 9 languages in the SPMRL dataset.

Motivation & Objective

• Show that a self-attentive encoder can outperform an LSTM encoder in a state-of-the-art discriminative constituency parser.
• Investigate how attention types (content vs. position) affect parsing performance and explore explicit factoring of information.
• Evaluate different lexical representations, including subword features and pre-trained contextual embeddings, for parsing accuracy.
• Demonstrate state-of-the-art results on WSJ without external data and with pre-trained representations; assess multilingual generalization on SPMRL.
• Analyze the role of long-distance dependencies and global context in parsing decisions.

Proposed method

• Use an encoder-decoder architecture where the encoder is a self-attentive network inspired by the Transformer, with 8 layers of multi-headed self-attention and position-wise feed-forward sublayers.
• Compute span scores s(i,j,l) via a neural network over pairwise endpoint representations, following the approach of Stern et al. (2017a).
• Input representations combine word embeddings, POS tag embeddings, and learned position embeddings; extend with factorized (separated) content and position information.
• Train with a margin-based hinge loss over labeled spans, using CKY-style inference for decoding.
• Experiment with factored attention to separate content and position signals, and with alternative lexical representations (CharLSTM, CharConcat, ELMo).
• Evaluate on Penn Treebank WSJ and the SPMRL multilingual dataset; report development and test set F1 scores.

Experimental results

Research questions

• RQ1 Can a self-attentive encoder outperform an LSTM-based encoder for discriminative constituency parsing?
• RQ2 Does explicitly separating content and position information in the encoder improve parsing accuracy?
• RQ3 What is the impact of different lexical representations, including subword features and contextual embeddings, on parsing performance?
• RQ4 How well does the model with self-attentive encoder perform on WSJ and across languages in SPMRL?

Key findings

• Self-attentive encoder improves over an LSTM encoder on WSJ development data (92.67 F1 vs 92.24 F1).
• Factoring content and position information yields performance gains (93.15 F1 on development with factoring).
• Disabling content attention has limited impact, with position attention being more critical overall; content helps more in later layers.
• Long-distance attention is essential for maximum accuracy; strict windowing harms performance, while relaxed windowing preserves some global pooling benefits.
• CharLSTM lexical representations outperform POStags and even enable good results without word embeddings (CharLSTM 93.61 dev with embeddings; 93.40 with no embeddings).
• ELMo contextual embeddings boost WSJ test F1 to 95.13 (state of the art for single-system WSJ) with a 4-layer encoder; base 93.55 F1 (single system, WSJ only) achieved without external data.