[Paper Review] Humans and transformer LMs: Abstraction drives language learning

Jasper Jian, Christopher D. Manning|arXiv (Cornell University)|Mar 18, 2026

Language Development and Disorders0 citations

TL;DR

The paper shows that GPT-2 pretrained transformers learn linguistic abstractions for verb classes and syntactic constructions earlier than item-specific patterns, supporting an abstraction-first learning bias over exemplar-based accounts.

ABSTRACT

Categorization is a core component of human linguistic competence. We investigate how a transformer-based language model (LM) learns linguistic categories by comparing its behaviour over the course of training to behaviours which characterize abstract feature-based and concrete exemplar-based accounts of human language acquisition. We investigate how lexical semantic and syntactic categories emerge using novel divergence-based metrics that track learning trajectories using next-token distributions. In experiments with GPT-2 small, we find that (i) when a construction is learned, abstract class-level behaviour is evident at earlier steps than lexical item-specific behaviour, and (ii) that different linguistic behaviours emerge abruptly in sequence at different points in training, revealing that abstraction plays a key role in how LMs learn. This result informs the models of human language acquisition that LMs may serve as an existence proof for.

Motivation & Objective

• Investigate how transformer LMs acquire linguistic categories during pretraining.
• Compare LM learning trajectories to abstraction-first vs exemplar-first human language acquisition accounts.
• Track emergence of lexical semantic and syntactic categories using divergence-based metrics over next-token distributions.

Proposed method

• Use GPT-2 small models trained on OpenWebText with 450 checkpoints to monitor learning trajectories.
• Define pairwise divergence (D_JS) between next-token distributions to assess class-general vs item-specific learning.
• Compare LM trajectories to abstraction-first and exemplar-first predictions using novel divergence-based metrics.
• Evaluate both semantic/argument-structure learning and syntactic subcategorization/non-local dependencies.
• Include an exemplar-first baseline using count-based co-occurrence vectors to contrast learning patterns.

Experimental results

Research questions

• RQ1 Do LM learning trajectories exhibit early abstraction over classes rather than item-specific patterns?
• RQ2 Are linguistic constructions learned in a sequential, abstraction-first manner across phenomena like verb classes, transitivity, and filler-gap dependencies?
• RQ3 Does an exemplar-first baseline account for LM learning as observed in training trajectories?
• RQ4 What is the ordering of abstraction emergence across different linguistic phenomena in LMs?

Key findings

• Abstraction-first behavior emerges across multiple phenomena, with class-level distinctions appearing early before item-level learning.
• Linguistic constructions are learned for classes of words before individual lexical items, and the onset of learning varies by phenomenon.
• Learning trajectories show sequential emergence of abstractions (semantic verb classes, then transitivity, then relative clause patterns).
• An exemplar-first baseline does not replicate GPT-2 learning trajectories, supporting abstraction-driven learning over exemplar memorization.
• Relative clause learning shows class-based emergence but with some class-specific timing differences, indicating structured abstraction rather than uniform parallelism.
• The observed abstraction-first patterns align with certain human language acquisition theories, while noting LMs rely on distributional data rather than innate abstractions.