[Paper Review] Grammar as a Foreign Language

Oriol Vinyals, Łukasz Kaiser+4 more|arXiv (Cornell University)|Dec 23, 2014

Natural Language Processing Techniques31 references402 citations

TL;DR

This paper proposes a domain-agnostic, attention-enhanced sequence-to-sequence LSTM model for syntactic constituency parsing, treating parse trees as linearized sequences. It achieves state-of-the-art F1 scores of 92.5 (single model) and 92.8 (ensemble) on the WSJ test set by training on a large synthetic corpus generated by existing parsers, outperforming both standard parsers and the BerkeleyParser, while also demonstrating high data efficiency and speed.

ABSTRACT

Syntactic constituency parsing is a fundamental problem in natural language processing and has been the subject of intensive research and engineering for decades. As a result, the most accurate parsers are domain specific, complex, and inefficient. In this paper we show that the domain agnostic attention-enhanced sequence-to-sequence model achieves state-of-the-art results on the most widely used syntactic constituency parsing dataset, when trained on a large synthetic corpus that was annotated using existing parsers. It also matches the performance of standard parsers when trained only on a small human-annotated dataset, which shows that this model is highly data-efficient, in contrast to sequence-to-sequence models without the attention mechanism. Our parser is also fast, processing over a hundred sentences per second with an unoptimized CPU implementation.

Motivation & Objective

• To investigate whether generic sequence-to-sequence models with attention can achieve state-of-the-art performance in syntactic constituency parsing without task-specific architectural constraints.
• To evaluate the data efficiency of attention-based sequence-to-sequence models compared to standard sequence-to-sequence models on small human-annotated parsing datasets.
• To assess the utility of synthetic, automatically labeled datasets—generated by existing parsers—for training high-performing parsing models.
• To determine whether models trained on imperfectly labeled synthetic data can surpass the performance of the parsers used to generate the training data.
• To develop a fast, scalable, and domain-agnostic parsing system that matches or exceeds the accuracy of complex, hand-crafted, domain-specific parsers.

Proposed method

• The model uses a deep Long Short-Term Memory (LSTM) network to encode the input sentence and decode the linearized parse tree, with a shared parameterization for input and output sequences.
• An attention mechanism is applied during decoding, allowing the model to dynamically attend to relevant encoder states at each decoding step, improving alignment for long sequences.
• The model is trained via stochastic gradient descent to maximize the log-likelihood of the correct linearized parse tree given the input sentence.
• Synthetic training data is generated using the BerkeleyParser on a large corpus, with high-confidence parse trees selected via agreement between two parsers to improve data quality.
• The model is fine-tuned on small human-annotated datasets (e.g., 1M tokens) to evaluate data efficiency and generalization.
• Inference is performed autoregressively, generating tokens one at a time until an end-of-sequence token is produced.

Experimental results

Research questions

• RQ1 Can a generic sequence-to-sequence model with attention achieve state-of-the-art performance in syntactic constituency parsing without architectural modifications for parsing?
• RQ2 How does the data efficiency of attention-enhanced sequence-to-sequence models compare to standard sequence-to-sequence models on small human-annotated parsing datasets?
• RQ3 Can synthetic datasets with potentially noisy labels from existing parsers still yield superior parsing performance compared to the original parsers?
• RQ4 Does the model’s ability to ignore errors in teacher-generated labels contribute to improved performance on the test set?
• RQ5 Can a single, non-ensemble model trained on synthetic data surpass the performance of a widely used, domain-specific parser like the BerkeleyParser?

Key findings

• The attention-enhanced sequence-to-sequence model achieved an F1 score of 92.5 on the WSJ test set (section 23) when trained on a synthetic corpus of high-confidence parse trees, setting a new state-of-the-art.
• With an ensemble, the model reached an F1 score of 92.8, further surpassing the BerkeleyParser’s performance of 90.4 on the same test set.
• On a small human-annotated dataset (1M tokens), the model achieved an F1 score of 88.3 without ensembling and 90.5 with ensembling, matching the BerkeleyParser’s 90.4 F1 score.
• The model processed over 100 sentences per second using an unoptimized CPU implementation, demonstrating high inference speed.
• The model outperformed the BerkeleyParser even though it was trained on data generated by the BerkeleyParser, suggesting that the student model effectively learned to ignore labeling noise.
• The attention mechanism significantly improved data efficiency, enabling strong performance on small human-annotated datasets where standard sequence-to-sequence models failed.