[Paper Review] Recurrent babbling: evaluating the acquisition of grammar from limited input data

Ludovica Pannitto, Aurélie Herbelot|arXiv (Cornell University)|Oct 9, 2020

Natural Language Processing Techniques79 references14 citations

TL;DR

This paper evaluates grammar acquisition in a character-level LSTM trained on a realistic 3M-word subset of child-directed speech from the CHILDES corpus. Using a novel usage-based methodology that tracks distributional shifts in generated 'babbling' over time, the authors demonstrate that the network progressively abstracts and reproduces grammatical constructions, including non-lexicalized syntactic patterns, indicating emergent productivity and compositional generalization from limited input.

ABSTRACT

Recurrent Neural Networks (RNNs) have been shown to capture various aspects of syntax from raw linguistic input. In most previous experiments, however, learning happens over unrealistic corpora, which do not reflect the type and amount of data a child would be exposed to. This paper remedies this state of affairs by training a Long Short-Term Memory network (LSTM) over a realistically sized subset of child-directed input. The behaviour of the network is analysed over time using a novel methodology which consists in quantifying the level of grammatical abstraction in the model's generated output (its "babbling"), compared to the language it has been exposed to. We show that the LSTM indeed abstracts new structuresas learning proceeds.

Motivation & Objective

• To investigate whether RNNs can learn abstract grammatical structures from limited, child-like input data, challenging assumptions from traditional generative grammar.
• To develop a methodology that evaluates grammatical abstraction in neural networks without relying on downstream tasks or external annotations.
• To test the hypothesis that neural networks can simulate human-like grammatical productivity through a usage-based, constructionist lens, blurring the line between lexicon and syntax.
• To assess how distributional patterns in model-generated language evolve over training, using cosine similarity between catenae in input and output spaces.

Proposed method

• Train a vanilla character-level LSTM on a 3M-word subset of CHILDES, focusing on child-directed English.
• Generate text samples ('babbling') at multiple training epochs to track linguistic evolution.
• Use dependency parsing to extract catenae (syntactic constituents) from both input and generated output.
• Compute cosine similarity between catenae in input (CHILDES) and model-generated (babbling) distributions to quantify distributional shift.
• Apply Dunn post-hoc tests to compare similarity shifts across three groups: negative, intermediate, and positive shift in similarity over time.
• Use distributional semantics to model form-meaning pairs as constructions, treating all items—lexical, partial, or syntactic—as equivalent units in a constructicon.

Experimental results

Research questions

• RQ1 To what extent does a character-level LSTM trained on limited child-directed input abstract and reproduce grammatical constructions over time?
• RQ2 How do distributional similarities between input and model-generated language evolve, and what does this reveal about syntactic abstraction?
• RQ3 Can neural networks simulate the productivity and compositionality observed in human language acquisition, even without explicit inductive bias?
• RQ4 How do different types of input data (e.g., OpenSubtitles vs. Simple Wikipedia) affect the model’s ability to learn and generalize grammatical patterns?

Key findings

• The LSTM model exhibits a significant increase in distributional similarity between its generated 'babbling' and the input CHILDES data over training, with a mean cosine similarity shift of 0.18 for high-shift catenae.
• Catenae such as 'you VERB you' and 'we can VERB' show a 0.49 to 0.74 increase in cosine similarity from epoch 5 to 35, indicating growing alignment with input patterns.
• The model successfully reproduces non-lexicalized syntactic patterns, such as 'AUX hungry @cop @conj', with a cosine similarity of 0.68 at epoch 35, suggesting abstraction beyond rote memorization.
• OpenSubtitles data shows higher similarity to CHILDES than Simple Wikipedia, suggesting it is a more suitable benchmark for studying child-like grammar acquisition.
• The Dunn post-hoc test confirms that the shift in similarity is statistically significant (p < 0.05) for 6.83e-06 and 4.15e-29 in the negative and positive groups, respectively, validating the robustness of the observed trends.
• The model’s ability to generalize to unlexicalized constructions—such as 'what @advmod VERB'—demonstrates emergent productivity, supporting the hypothesis of compositional generalization in RNNs.