[Paper Review] Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks

R. Thomas McCoy, Robert Frank+1 more|arXiv (Cornell University)|Feb 25, 2018

Natural Language Processing Techniques19 references47 citations

TL;DR

The paper tests whether recurrent neural networks can learn hierarchical question formation without inherent hierarchical biases, finding that one GRU with attention generalizes hierarchically, especially with an agreement cue in the input.

ABSTRACT

Syntactic rules in natural language typically need to make reference to hierarchical sentence structure. However, the simple examples that language learners receive are often equally compatible with linear rules. Children consistently ignore these linear explanations and settle instead on the correct hierarchical one. This fact has motivated the proposal that the learner's hypothesis space is constrained to include only hierarchical rules. We examine this proposal using recurrent neural networks (RNNs), which are not constrained in such a way. We simulate the acquisition of question formation, a hierarchical transformation, in a fragment of English. We find that some RNN architectures tend to learn the hierarchical rule, suggesting that hierarchical cues within the language, combined with the implicit architectural biases inherent in certain RNNs, may be sufficient to induce hierarchical generalizations. The likelihood of acquiring the hierarchical generalization increased when the language included an additional cue to hierarchy in the form of subject-verb agreement, underscoring the role of cues to hierarchy in the learner's input.

Motivation & Objective

• Assess whether non-hierarchically biased RNNs can learn hierarchical question formation from limited data.
• Evaluate multiple RNN architectures on language fragments with and without subject–verb agreement.
• Investigate how input cues to hierarchy affect the emergence of hierarchical generalization.
• Analyze how architecture type and initialization influence generalization behavior.

Proposed method

• Use sequence-to-sequence RNNs (encoder–decoder) to model declarative sentences and their question forms.
• Test six architectures: SRN, GRU, LSTM, each with and without attention, across two language fragments (no-agreement and agreement).
• Train 100 networks per architecture (1200 total) on 120,000 sentences; evaluate on a 10,000-sentence test set and a 10,000-sentence generalization set.
• Two tasks are trained: identity (IDENT) and question formation (QUEST); the generalization set contains withheld sentence types that disambiguate linear vs hierarchical hypotheses.
• Assess whether predictions align with hierarchical vs linear rules by inspecting the first output auxiliary in the generalization set when linearly and hierarchically distinct.

Experimental results

Research questions

• RQ1 Can GRU/LSTM/GRU-with-attention networks learn hierarchical subject-auxiliary inversion without an explicit hierarchical bias?
• RQ2 Does providing hierarchical cues (subject–verb agreement) increase the likelihood of hierarchical generalization?
• RQ3 How do different RNN architectures and initializations affect hierarchical generalization outcomes?
• RQ4 What errors do networks make compared to human syntactic generalization errors, and what do these reveal about learning biases?

Key findings

• All six architectures except the vanilla SRN achieved over 94% exact accuracy on the test set; the best was 99.9% (LSTM without attention).
• On the generalization set, only about 13% of questions were exactly correct in the best architecture (GRU with attention).
• Adding agreement in the input increased the probability of hierarchical generalization across architectures.
• Initialization varied accuracy for each architecture, indicating biases are not uniformly strong across random starts.
• The GRU with attention showed a qualitative shift toward hierarchical generalization, unlike other architectures where linear rules dominated; attention made a difference for GRU.concat
• GRU with attention encoded information beyond linear order, suggesting its generalization relied on hierarchical cues rather than purely linear representations.