[Paper Review] Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times

Byung-Doh Oh, Shisen Yue+1 more|arXiv (Cornell University)|Feb 3, 2024

Topic Modeling4 citations

TL;DR

This paper identifies word frequency as the key factor explaining why larger language models (LLMs) and those trained on more data show reduced fit between their surprisal estimates and human reading times. Using analyses across four model families and corpora, it demonstrates that larger models overfit to rare words by learning superhumanly complex associations, leading to worse prediction of human reading behavior, especially for low-frequency words.

ABSTRACT

Recent studies have shown that as Transformer-based language models become larger and are trained on very large amounts of data, the fit of their surprisal estimates to naturalistic human reading times degrades. The current work presents a series of analyses showing that word frequency is a key explanatory factor underlying these two trends. First, residual errors from four language model families on four corpora show that the inverse correlation between model size and fit to reading times is the strongest on the subset of least frequent words, which is driven by excessively accurate predictions of larger model variants. Additionally, training dynamics reveal that during later training steps, all model variants learn to predict rare words and that larger model variants do so more accurately, which explains the detrimental effect of both training data amount and model size on fit to reading times. Finally, a feature attribution analysis demonstrates that larger model variants are able to accurately predict rare words based on both an effectively longer context window size as well as stronger local associations compared to smaller model variants. Taken together, these results indicate that Transformer-based language models' surprisal estimates diverge from human-like expectations due to the superhumanly complex associations they learn for predicting rare words.

Motivation & Objective

• To investigate why larger language models and those trained on more data show a declining fit between their surprisal estimates and human reading times.
• To determine whether word frequency modulates the inverse correlation between model size/training data and surprisal's predictive power.
• To examine how training dynamics and model architecture influence the prediction of rare versus frequent words.
• To assess whether the over-accuracy on rare words in large models undermines their ability to model human processing difficulty.
• To explore whether frequency effects and predictability effects are dissociable in naturalistic reading comprehension.

Proposed method

• Evaluated surprisal estimates from four Transformer-based language model families (e.g., GPT, BERT variants) on four naturalistic reading time corpora: Natural Stories, Dundee, GECO, and Provo.
• Computed residual errors between model-predicted surprisal and observed reading times, stratified by word frequency quantiles.
• Tracked training dynamics across model variants to analyze how and when rare words are learned with increasing training tokens.
• Applied feature attribution techniques (e.g., gradient-based or attention ablation) to isolate contributions of context window length and local attention strength to rare word prediction.
• Used regression models to assess the fit of surprisal estimates to reading times, with and without frequency-based stratification.
• Compared model variants of different sizes (small, medium, large) trained on increasing amounts of data (up to 2B tokens) to isolate size and data effects.

Figure 1: Corpus-level perplexity measures from each GPT-2, GPT-Neo, OPT, and Pythia model variant, and mean squared errors of regression models that include each surprisal predictor on the four corpora of reading times. The ordered labels represent variants of different sizes, where ‘1’ represents

Experimental results

Research questions

• RQ1 Does the inverse correlation between model size and surprisal’s fit to reading times vary by word frequency?
• RQ2 How do training dynamics influence the learning of rare word predictions across model variants of different sizes?
• RQ3 To what extent do larger models learn more accurate representations for rare words compared to smaller models?
• RQ4 What mechanisms (e.g., context window length, attention strength) underlie the superior rare word prediction in larger models?
• RQ5 Can the divergence between LMs’ surprisal and human reading times be explained by overfitting to frequency-irrelevant, complex associations in rare words?

Key findings

• The inverse correlation between model size and fit to reading times is strongest for the least frequent words, where larger models produce significantly more accurate surprisal estimates.
• Larger model variants learn to predict rare words more accurately than smaller ones, especially during later training stages, which drives the degradation in fit to human reading times.
• Training dynamics show that all model variants begin learning rare word predictions only after substantial exposure (e.g., beyond 2B tokens), with larger models achieving higher accuracy more quickly.
• Feature attribution analysis reveals that larger models predict rare words more accurately due to both an effectively longer context window and stronger local attention associations.
• The over-accuracy on rare words—driven by superhumanly complex associations—explains why surprisal from large models diverges from human-like processing expectations.
• Limiting context window size or weakening local associations improves the fit of surprisal to reading times, confirming that overfitting to rare word patterns harms human processing prediction.

Figure 2: Corpus-level perplexity measures from each GPT-2, GPT-Neo, OPT, and Pythia model variant, and mean squared errors of regression models that include each surprisal predictor on the four corpora of reading times. The columns of subplots represent the five quintiles defined by unigram log-pro