[Paper Review] Linguistic Analysis of Pretrained Sentence Encoders with Acceptability Judgments

Alex Warstadt, Samuel R. Bowman|arXiv (Cornell University)|Jan 11, 2019

Topic ModelingComputer Science34 references19 citations

TL;DR

This paper introduces a fine-grained, linguistically annotated evaluation set for pretrained sentence encoders by labeling 1,043 sentences from the CoLA development set with 13 syntactic phenomena. Using this dataset, the authors probe BERT, GPT, and a BiLSTM baseline, finding that while all models handle complex argument structures well, long-distance dependencies like questions remain challenging—especially for the BiLSTM, which lags significantly behind BERT and GPT.

ABSTRACT

Recent work on evaluating grammatical knowledge in pretrained sentence encoders gives a fine-grained view of a small number of phenomena. We introduce a new analysis dataset that also has broad coverage of linguistic phenomena. We annotate the development set of the Corpus of Linguistic Acceptability (CoLA; Warstadt et al., 2018) for the presence of 13 classes of syntactic phenomena including various forms of argument alternations, movement, and modification. We use this analysis set to investigate the grammatical knowledge of three pretrained encoders: BERT (Devlin et al., 2018), GPT (Radford et al., 2018), and the BiLSTM baseline from Warstadt et al. We find that these models have a strong command of complex or non-canonical argument structures like ditransitives (Sue gave Dan a book) and passives (The book was read). Sentences with long distance dependencies like questions (What do you think I ate?) challenge all models, but for these, BERT and GPT have a distinct advantage over the baseline. We conclude that recent sentence encoders, despite showing near-human performance on acceptability classification overall, still fail to make fine-grained grammaticality distinctions for many complex syntactic structures.

Motivation & Objective

• To address the lack of fine-grained, domain-general linguistic evaluation for pretrained sentence encoders by creating a dataset that links acceptability judgments to specific syntactic phenomena.
• To investigate how well BERT, GPT, and a BiLSTM baseline understand diverse syntactic structures, including argument alternations, movement, and modification.
• To identify which syntactic phenomena pose the greatest challenges for current models, especially in terms of long-distance dependencies and non-canonical structures.
• To provide a linguistically interpretable benchmark that enables direct comparison between model performance and theoretical linguistic concepts.

Proposed method

• Annotating the CoLA development set with 13 syntactic phenomenon classes, each defined as a union of more specific constructions, using expert linguistic knowledge from mainstream publications.
• Using the annotated dataset as a probing task: training a small classifier on top of pretrained sentence encoders (BERT, GPT, BiLSTM) to predict acceptability.
• Evaluating model performance on specific syntactic classes, comparing accuracy across phenomena to identify relative strengths and weaknesses.
• Focusing on phenomena such as passives, clefts, questions, ellipsis, and adjuncts to assess sensitivity to syntactic complexity and structural distance.
• Using the original CoLA dataset as a source of acceptability judgments, while enriching it with detailed syntactic annotations to enable fine-grained analysis.
• Applying standard probing methodology to isolate grammatical knowledge from general representation learning, treating acceptability prediction as a diagnostic task.

Experimental results

Research questions

• RQ1 Which syntactic phenomena are most challenging for pretrained sentence encoders like BERT and GPT, and how do they compare to a BiLSTM baseline?
• RQ2 To what extent do BERT and GPT demonstrate superior grammatical knowledge compared to the BiLSTM on phenomena involving long-distance dependencies, such as questions and clefts?
• RQ3 How do non-canonical argument structures like passives and ditransitives affect model performance, and do they pose significant challenges?
• RQ4 Are there specific syntactic constructions where BERT and GPT show no advantage over the BiLSTM, and what does this imply about their linguistic generalization?
• RQ5 How well do these models distinguish between grammatical acceptability and subtle syntactic violations, such as agreement or word order errors?

Key findings

• BERT and GPT achieve significantly higher accuracy than the BiLSTM on movement phenomena such as clefts (e.g., 'It is Bo that left') and questions with long-distance dependencies (e.g., 'What do you think I ate?').
• The BiLSTM model struggles most with long-distance dependencies, while BERT and GPT maintain strong performance, indicating their attention mechanisms better handle structural complexity.
• Non-canonical argument structures like passives ('The book was read') and ditransitives ('Sue gave Dan a book') are well understood by all models, suggesting robust learning of argument alternation patterns.
• Constructions with adjuncts (e.g., 'Sue exercises in the morning') show no significant performance difference between models, indicating that these are not particularly challenging for any of the encoders.
• Ellipsis and anaphora (e.g., 'I saw Bill while you did so Mary') are difficult for all models, but BERT and GPT show a modest advantage over the BiLSTM on these structures.
• The study reveals that despite near-human performance on overall acceptability classification, models still fail to make fine-grained grammaticality distinctions on complex syntactic structures, especially those involving long-distance dependencies.