[Paper Review] Distinct patterns of syntactic agreement errors in recurrent networks and humans

Tal Linzen, Brian Leonard|arXiv (Cornell University)|Jul 18, 2018

Neurobiology of Language and Bilingualism8 references42 citations

TL;DR

The paper compares how RNNs and humans error on subject-verb agreement across attractors and syntactic structures, revealing both similarities and crucial differences that suggest RNNs’ representations diverge from human syntactic processing.

ABSTRACT

Determining the correct form of a verb in context requires an understanding of the syntactic structure of the sentence. Recurrent neural networks have been shown to perform this task with an error rate comparable to humans, despite the fact that they are not designed with explicit syntactic representations. To examine the extent to which the syntactic representations of these networks are similar to those used by humans when processing sentences, we compare the detailed pattern of errors that RNNs and humans make on this task. Despite significant similarities (attraction errors, asymmetry between singular and plural subjects), the error patterns differed in important ways. In particular, in complex sentences with relative clauses error rates increased in RNNs but decreased in humans. Furthermore, RNNs showed a cumulative effect of attractors but humans did not. We conclude that at least in some respects the syntactic representations acquired by RNNs are fundamentally different from those used by humans.

Motivation & Objective

• Investigate whether RNNs’ syntactic representations resemble those used by humans during sentence processing.
• Examine how attractors and their structural position affect agreement errors in both RNNs and humans.
• Test whether error patterns differ between PP and RC modifiers to infer underlying representations.
• Assess whether humans and RNNs show cumulative attraction effects with multiple attractors.

Proposed method

• Train 20 LSTM-based RNNs (50 and 1000 hidden units) on verb-number prediction from preambles created from Wikipedia sentences.
• Use preambles with a deleted verb to elicit singular/plural agreement decisions (is/are) under three time-pressure paradigms: RSVP, SPR, and Untimed.
• Compare human participants’ errors in Experiment 1 to RNN predictions on identical materials to identify qualitative similarities and differences.
• Conduct Experiment 2 with RC and PP constructions to test cumulative attraction and RC-length heuristics.
• Simulate human results with 1000-dimensional RNNs and manipulate RC length and word order to probe the short RC heuristic.
• Provide code and materials for replication at the authors’ GitHub repository.

Experimental results

Research questions

• RQ1 Do RNNs and humans show attraction effects when an attractor differs in number from the subject?
• RQ2 How do the structural positions of attractors (PP vs RC) influence agreement errors in humans vs RNNs?
• RQ3 Do RNNs exhibit cumulative attraction with multiple attractors, and is this mirrored in humans?
• RQ4 Are there qualitative differences in error patterns between PP and RC modifiers that reveal different underlying representations?

Key findings

• Humans show attraction errors that increase when a local noun differs in number from the subject.
• Error asymmetry: attraction errors are more frequent when the subject is singular and the attractor plural.
• Experiment 1 shows that PP attractors yield more errors than RC attractors for humans, while RNNs exhibit more errors in RCs than PPs.
• RNNs exhibit a cumulative attraction effect with multiple attractors, unlike humans in Experiment 2.
• Reversing RC order in materials increases RNN errors, supporting a short RC heuristic rather than purely proximity-based interference.