[Paper Review] Falling Through the Gaps: Neural Architectures as Models of Morphological Rule Learning

Deniz Beser|arXiv (Cornell University)|May 8, 2021

Natural Language Processing Techniques4 citations

TL;DR

This paper evaluates the Transformer and RNN architectures as models of morphological rule learning, focusing on their ability to generalize productive inflectional patterns. Despite the Transformer's superior performance on standard tasks, both models produce inflected forms for Russian verbs that native speakers do not use—demonstrating they fail to capture linguistic gaps, indicating a need for structural priors in neural models of language learning.

ABSTRACT

Recent advances in neural architectures have revived the problem of morphological rule learning. We evaluate the Transformer as a model of morphological rule learning and compare it with Recurrent Neural Networks (RNN) on English, German, and Russian. We bring to the fore a hitherto overlooked problem, the morphological gaps, where the expected inflection of a word is missing. For example, 63 Russian verbs lack a first-person-singular present form such that one cannot comfortably say "*o\\v{s}\\v{c}u\\v{s}\\v{c}u" ("I feel"). Even English has gaps, such as the past participle of "stride": the function of morphological inflection can be partial. Both neural architectures produce inflections that ought to be missing. Analyses reveal that Transformers recapitulate the statistical distribution of inflections in the training data, similar to RNNs. Models' success on English and German is driven by the fact that rules in these languages can be identified with the majority forms, which is not universal.

Motivation & Objective

• To assess whether the Transformer architecture, as a state-of-the-art neural model, can serve as a better cognitive model of morphological rule learning than RNNs.
• To investigate whether neural architectures can learn productive morphological rules that reflect human-like generalization, especially in cases of non-majority productivity and morphological gaps.
• To examine whether models exhibit sensitivity to distributional patterns in training data, leading to analogical errors or overgeneralization, even when such forms are not native-like.
• To evaluate whether models can implicitly learn to reject inflection for words with known morphological gaps, as humans do.
• To propose morphological gaps as a diagnostic tool for evaluating the cognitive plausibility of neural language models.

Proposed method

• Trained both Transformer and RNN sequence-to-sequence models on morphological inflection data from English, German, and Russian.
• Used a standard training set and a special test set containing 63 Russian verbs that lack a first-person-singular present tense form (morphological gaps).
• Measured model confidence via likelihood scores of predicted inflected forms to assess whether models struggle with gapped verbs.
• Compared model predictions across regular verbs and gapped verbs to detect differences in confidence, indicating implicit rejection capability.
• Analyzed model behavior across different linguistic phenomena: English past tense (majority rule), German plural (non-majority productivity), and Russian gaps (structural absence).
• Used statistical correlation and similarity metrics to compare predictions between Transformer and RNN models, assessing consistency across conditions.

Experimental results

Research questions

• RQ1 Can the Transformer architecture generalize morphological rules in a way that aligns with human-like productivity, especially in cases where the productive rule is not the majority form?
• RQ2 Do neural architectures like Transformers and RNNs exhibit the same analogical and distributional biases as observed in earlier models, particularly in their overgeneralization of inflections?
• RQ3 Do models show reduced confidence or increased uncertainty when predicting inflections for verbs that are known to have morphological gaps in native speech?
• RQ4 To what extent do both models recapitulate the statistical distribution of inflectional forms in training data, even when such forms are not used by native speakers?
• RQ5 Can morphological gaps be used as a diagnostic benchmark to evaluate the cognitive plausibility of neural sequence models in morphology?

Key findings

• Both the Transformer and RNN models produce first-person-singular present tense inflections for 63 Russian verbs that native speakers do not use, indicating they fail to respect morphological gaps.
• Model confidence did not significantly drop on the gapped verb test set: RNN confidence decreased from 0.95 (σ=0.12) to 0.86 (σ=0.18), and Transformer from 0.46 (σ=0.11) to 0.39 (σ=0.13), showing no meaningful distinction between regular and gapped verbs.
• The models' predictions were highly correlated across both architectures, indicating similar behavior in distributional generalization, regardless of architectural differences.
• Despite the Transformer’s stronger performance on standard NLP tasks, it is no more effective than RNNs at modeling human-like morphological rule learning, particularly in cases of non-majority productivity or structural absence.
• The models’ behavior suggests they learn from statistical distribution of inflections rather than from underlying linguistic structure, leading to overgeneralization in the same way as RNNs.
• The study demonstrates that morphological gaps are a valid diagnostic for evaluating cognitive plausibility in neural language models, as both architectures fail to reflect native speaker constraints.