[Paper Review] Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments.

Alex Warstadt, Samuel R. Bowman|arXiv (Cornell University)|Jan 11, 2019

Natural Language Processing TechniquesComputer Science29 references22 citations

TL;DR

This paper evaluates the syntactic knowledge of pretrained sentence encoders—OpenAI GPT, BERT, and others—by fine-tuning them on a grammatically annotated development set from the Corpus of Linguistic Acceptability (CoLA). Results show that while models easily learn simple syntactic phenomena like adjunct modification, they struggle with complex structures such as long-distance dependencies and morphological agreement, indicating limited implicit syntactic understanding despite strong performance on downstream tasks.

ABSTRACT

Recent pretrained sentence encoders achieve state of the art results on language understanding tasks, but does this mean they have implicit knowledge of syntactic structures? We introduce a grammatically annotated development set for the Corpus of Linguistic Acceptability (CoLA; Warstadt et al., 2018), which we use to investigate the grammatical knowledge of three pretrained encoders, including the popular OpenAI Transformer (Radford et al., 2018) and BERT (Devlin et al., 2018). We fine-tune these encoders to do acceptability classification over CoLA and compare the models' performance on the annotated analysis set. Some phenomena, e.g. modification by adjuncts, are easy to learn for all models, while others, e.g. long-distance movement, are learned effectively only by models with strong overall performance, and others still, e.g. morphological agreement, are hardly learned by any model.

Motivation & Objective

• To assess whether pretrained sentence encoders implicitly learn syntactic structures despite their success on NLP tasks.
• To investigate the extent to which models can learn complex syntactic phenomena such as long-distance movement and morphological agreement.
• To develop and use a grammatically annotated development set for CoLA to enable systematic evaluation of syntactic generalization.
• To compare the performance of multiple state-of-the-art encoders—GPT, BERT, and others—on syntactic acceptability classification.

Proposed method

• Fine-tune three pretrained sentence encoders—OpenAI GPT, BERT, and a third model—on a grammatically annotated subset of the Corpus of Linguistic Acceptability (CoLA).
• Use the annotated development set to evaluate model performance on various syntactic phenomena, including adjunct modification, long-distance movement, and morphological agreement.
• Train a binary classifier for acceptability judgments, where inputs are sentences and outputs are human-annotated acceptability scores.
• Analyze model predictions across syntactic categories to determine which structures are learned effectively and which are not.
• Compare performance across models to identify differences in syntactic generalization, especially between high-performing and lower-performing models.
• Use quantitative metrics to evaluate classification accuracy on distinct syntactic phenomena, enabling fine-grained analysis of syntactic knowledge.

Experimental results

Research questions

• RQ1 To what extent do pretrained sentence encoders learn syntactic structures such as adjunct modification and long-distance dependencies?
• RQ2 How well do these models generalize to morphological agreement, a complex syntactic phenomenon?
• RQ3 Which syntactic phenomena are most easily learned by models, and which remain challenging despite strong overall performance?
• RQ4 Do models with higher overall performance on CoLA show better syntactic generalization across diverse syntactic constructions?

Key findings

• Models with strong overall performance on CoLA effectively learn long-distance movement, a complex syntactic phenomenon, while weaker models fail to do so.
• Adjunct modification is learned easily by all models, indicating that simpler syntactic structures are well-captured by current encoders.
• Morphological agreement is poorly learned by all models, suggesting a significant gap in their syntactic knowledge for such phenomena.
• The performance gap between models on syntactic phenomena correlates with their overall performance, indicating that syntactic generalization is not uniformly distributed across architectures.