[Paper Review] RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency

Richard Futrell, Ethan Wilcox+2 more|arXiv (Cornell University)|Sep 5, 2018

Topic ModelingComputer Science46 references40 citations

TL;DR

The paper empirically tests whether English and Japanese LSTM language models represent incremental syntactic state and grammatical dependencies, using controlled psycholinguistic experiments adapted to RNN surprisal measures. It finds evidence for incremental syntactic state and some maintenance of upcoming material, but shows limitations in learning reflexive binding and NPI licensing.

ABSTRACT

Recurrent neural networks (RNNs) are the state of the art in sequence modeling for natural language. However, it remains poorly understood what grammatical characteristics of natural language they implicitly learn and represent as a consequence of optimizing the language modeling objective. Here we deploy the methods of controlled psycholinguistic experimentation to shed light on to what extent RNN behavior reflects incremental syntactic state and grammatical dependency representations known to characterize human linguistic behavior. We broadly test two publicly available long short-term memory (LSTM) English sequence models, and learn and test a new Japanese LSTM. We demonstrate that these models represent and maintain incremental syntactic state, but that they do not always generalize in the same way as humans. Furthermore, none of our models learn the appropriate grammatical dependency configurations licensing reflexive pronouns or negative polarity items.

Motivation & Objective

• Assess whether RNNs maintain incremental syntactic state during processing of complex constructions.
• Evaluate RNNs' representations of grammatical dependencies such as reflexive binding and NPI licensing.
• Investigate garden-path effects and obligatory upcoming syntactic events to compare with human parsing.
• Compare English and Japanese RNN behavior to understand cross-language generalization of syntactic representations.

Proposed method

• Compute word-level surprisal from RNNs as the negative log probability of the next token given the previous hidden state.
• Design targeted sentence stimuli to elicit garden-path effects and measure surprisal differences across conditions.
• Preregister experiments and analyze by-item and by-condition surprisal with linear mixed-effects models.
• Test two English LSTMs (JRNN and GRNN) and one Japanese LSTM (JPRNN) trained on language modeling objectives.
• Conduct experiments on MV/RR garden-paths, subject animacy effects, obligatory upcoming syntactic events (ORCs and subordination), reflexive binding, and NPIs.

Experimental results

Research questions

• RQ1 Do LSTMs maintain incremental syntactic state in garden-path and reduced-relative contexts similar to humans?
• RQ2 Do English and Japanese LSTMs learn and apply grammatical dependencies such as reflexive binding and NPI licensing?
• RQ3 How do LSTMs handle obligatory upcoming syntactic events like completing relative clauses and subordinate clauses over time?
• RQ4 Are RNNs sensitive to fine-grained lexical-syntactic cues such as animacy in garden-pathing?
• RQ5 How do model architectures and training data influence the emergence of human-like syntactic representations?

Key findings

• LSTMs show evidence of incremental syntactic state within relative clauses and can use verb-form cues to signal reduced relative clauses.
• Both English models maintain expectations for upcoming material in ORCs and subordinations, with robustness that decreases as intervening material becomes longer or more complex.
• JRNN demonstrates reflexive pronoun gender mismatch effects consistent with learning some binding tendencies, but GRNN shows weaker or no such effects.
• Neither English model learns appropriate licensing configurations for reflexive binding or NPIs in English or Japanese, indicating gaps in learning grammatical dependencies.
• English NPIs show licensing effects, but models also exhibit spurious licensing from negative licensors in relative clauses, suggesting imperfect abstraction of licensing constraints.
• Japanese NPIs (shika) show partial licensing effects, with complex interactions indicating imperfect generalization of the NPI licensing rules.