[Paper Review] On the Effect of Anticipation on Reading Times

Tiago Pimentel, Clara Meister+3 more|arXiv (Cornell University)|Nov 25, 2022

Text Readability and Simplification4 citations

TL;DR

This paper investigates whether reading times are influenced by anticipatory processing—where readers predict upcoming words—rather than just reactive processing based on word surprisal. Using four naturalistic reading datasets, it shows that contextual entropy (a measure of prediction uncertainty) is a stronger predictor of reading times than surprisal in three of four datasets, with Rényi entropy (α=1/2) outperforming Shannon entropy, suggesting readers anticipate based on the number of plausible continuations rather than expected surprisal.

ABSTRACT

Over the past two decades, numerous studies have demonstrated how less predictable (i.e., higher surprisal) words take more time to read. In general, these studies have implicitly assumed the reading process is purely responsive: Readers observe a new word and allocate time to process it as required. We argue that prior results are also compatible with a reading process that is at least partially anticipatory: Readers could make predictions about a future word and allocate time to process it based on their expectation. In this work, we operationalize this anticipation as a word's contextual entropy. We assess the effect of anticipation on reading by comparing how well surprisal and contextual entropy predict reading times on four naturalistic reading datasets: two self-paced and two eye-tracking. Experimentally, across datasets and analyses, we find substantial evidence for effects of contextual entropy over surprisal on a word's reading time (RT): in fact, entropy is sometimes better than surprisal in predicting a word's RT. Spillover effects, however, are generally not captured by entropy, but only by surprisal. Further, we hypothesize four cognitive mechanisms through which contextual entropy could impact RTs -- three of which we are able to design experiments to analyze. Overall, our results support a view of reading that is not just responsive, but also anticipatory.

Motivation & Objective

• To investigate whether reading behavior is driven not only by reactive processing of words already seen, but also by anticipatory predictions about upcoming words.
• To test whether contextual entropy—measuring uncertainty in word predictions—predicts reading times more effectively than surprisal.
• To explore four cognitive mechanisms (word skipping, budgeting, preemptive processing, uncertainty cost) through which anticipation might influence reading times.
• To evaluate whether Rényi entropy with α=1/2 better captures readers' expectations than Shannon entropy.
• To determine whether anticipation effects are detectable in reading time data beyond what is explained by surprisal alone.

Proposed method

• Operationalized anticipation using contextual entropy, with Rényi entropy (α=1/2) as a measure of prediction uncertainty about future words.
• Compared surprisal and contextual entropy as predictors of reading times across four datasets: two self-paced reading and two eye-tracking datasets.
• Used linear mixed-effects models to assess predictive power of surprisal and entropy on reading times, controlling for word length and other covariates.
• Designed experiments to test four cognitive mechanisms of anticipation: word skipping, budgeting, preemptive processing, and uncertainty cost.
• Applied subadditivity and superadditivity properties of Rényi entropy to validate its theoretical robustness as a predictor.
• Analyzed dataset statistics and Spearman correlations between surprisal and entropy to rule out noise-based explanations for results.

Experimental results

Research questions

• RQ1 Does contextual entropy predict reading times better than surprisal, indicating anticipatory processing?
• RQ2 Are there specific cognitive mechanisms—such as word skipping or preemptive processing—through which anticipation influences reading times?
• RQ3 Does Rényi entropy with α=1/2 provide a better operationalization of anticipation than Shannon entropy?
• RQ4 Do spillover effects in reading times stem from surprisal rather than contextual entropy?
• RQ5 Is the predictive power of entropy independent of noise in surprisal estimates, as suggested by high correlation between surprisal and entropy?

Key findings

• Contextual entropy was a significant predictor of reading times in three out of four datasets, and in two of these, it outperformed surprisal as a predictor.
• Rényi entropy with α=1/2 consistently led to stronger predictive models than Shannon entropy, suggesting readers may anticipate based on the number of plausible word continuations.
• Spillover effects in reading times were captured by surprisal but not by contextual entropy, indicating entropy reflects anticipation rather than post-processing effects.
• The strong correlation between surprisal and entropy (shown in Figure 3) does not explain the superior predictive power of entropy, ruling out simple noise-averaging as the cause.
• The results support a reading model that is not purely responsive but also anticipatory, with readers allocating processing time based on expectations about upcoming words.
• The study provides empirical evidence that anticipation—particularly through uncertainty-based prediction—plays a measurable role in reading time allocation.