[Paper Review] Assessing BERT's Syntactic Abilities

Yoav Goldberg|arXiv (Cornell University)|Jan 16, 2019

Natural Language Processing Techniques6 references295 citations

TL;DR

The paper evaluates whether BERT's bidirectional, attention-based architecture captures English syntactic dependencies by adapting prior syntax tests to a masked, bidirectional setting; results show strong syntax sensitivity, with base often outperforming large.

ABSTRACT

I assess the extent to which the recently introduced BERT model captures English syntactic phenomena, using (1) naturally-occurring subject-verb agreement stimuli; (2) "coloreless green ideas" subject-verb agreement stimuli, in which content words in natural sentences are randomly replaced with words sharing the same part-of-speech and inflection; and (3) manually crafted stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT model performs remarkably well on all cases.

Motivation & Objective

• Motivate investigation into whether BERT captures syntax-sensitive dependencies in English.
• Adapt existing syntax evaluation stimuli to BERT's bidirectional masking framework.
• Compare BERT Base and BERT Large on standard syntactic phenomena.
• Discuss limitations and implications for purely attention-based models in syntax learning.

Proposed method

• Adapt stimuli from Linzen et al. (2016), Gulordava et al. (2018), and Marvin & Linzen (2018) to BERT's bidirectional setting.
• Mask the target verb and score predicted logits for correct versus incorrect inflections.
• Use complete sentences (excluding the verb) to evaluate predictions, comparing original vs alternate forms.
• Experiment with BERT-Base and BERT-Large under PyTorch implementation and pre-trained weights.
• Discard stimuli where the focus verb or its inflection are out-of-vocabulary or unsuitable for tokenization; omit certain NPI and copular cases as appropriate.

Experimental results

Research questions

• RQ1 Can BERT, with bidirectional masking, capture subject-verb agreement and related syntactic phenomena in English?
• RQ2 How do BERT Base and BERT Large compare on syntax-sensitive tasks?
• RQ3 What impact do dataset choices, pretraining corpora, and evaluation protocol have on observed syntactic performance?
• RQ4 Do results generalize beyond naturally occurring sentences to controlled or nonce stimuli?

Key findings

• BERT achieves high accuracy across syntactic test cases, often higher than previous LSTM-based models.
• For many conditions, BERT-Base outperforms BERT-Large rather than the opposite.
• Results are not directly comparable to prior work due to bidirectionality, different training data, and stimuli discards.
• Findings suggest purely attention-based models like BERT can capture hierarchy-sensitive syntax comparably to or better than LSTM models, given appropriate evaluation setups.
• Some exceptions exist (in a sentential complement and short VP coordination in Table 3), but overall trends favor strong syntactic generalization by BERT.
• The study emphasizes that larger models do not universally yield better syntactic performance.