[Paper Review] SEAM: An Integrated Activation-Coupled Model of Sentence Processing and Eye Movements in Reading

Maximilian M. Rabe, Dario Paape+3 more|arXiv (Cornell University)|Mar 9, 2023

Gaze Tracking and Assistive Technology85 references4 citations

TL;DR

SEAM is a novel computational model that integrates eye-movement control and sentence processing by coupling the SWIFT model of eye movements with the Lewis and Vasishth sentence comprehension framework. Using Bayesian inference via MCMC, SEAM successfully reproduces similarity-based interference effects in reading, marking the first integrated dynamical model of its kind in natural language comprehension.

ABSTRACT

Models of eye-movement control during reading, developed largely within psychology, usually focus on visual, attentional, lexical, and motor processes but neglect post-lexical language processing; by contrast, models of sentence comprehension processes, developed largely within psycholinguistics, generally focus only on post-lexical language processes. We present a model that combines these two research threads, by integrating eye-movement control and sentence processing. Developing such an integrated model is extremely challenging and computationally demanding, but such an integration is an important step toward complete mathematical models of natural language comprehension in reading. We combine the SWIFT model of eye-movement control (Seelig et al., 2020, doi:10.1016/j.jmp.2019.102313) with key components of the Lewis and Vasishth sentence processing model (Lewis & Vasishth, 2005, doi:10.1207/s15516709cog0000_25). This integration becomes possible, for the first time, due in part to recent advances in successful parameter identification in dynamical models, which allows us to investigate profile log-likelihoods for individual model parameters. We present a fully implemented proof-of-concept model demonstrating how such an integrated model can be achieved; our approach includes Bayesian model inference with Markov Chain Monte Carlo (MCMC) sampling as a key computational tool. The integrated Sentence-Processing and Eye-Movement Activation-Coupled Model (SEAM) can successfully reproduce eye movement patterns that arise due to similarity-based interference in reading. To our knowledge, this is the first-ever integration of a complete process model of eye-movement control with linguistic dependency completion processes in sentence comprehension. In future work, this proof of concept model will need to be evaluated using a comprehensive set of benchmark data.

Motivation & Objective

• To bridge the gap between eye-movement control models in psychology and sentence processing models in psycholinguistics.
• To develop a unified dynamical model that captures both oculomotor behavior and post-lexical linguistic processing during reading.
• To enable parameter identification in complex dynamical systems through profile log-likelihood analysis.
• To demonstrate feasibility of integrating full process models of eye movements and sentence comprehension using Bayesian MCMC methods.
• To provide a proof-of-concept model for future evaluation on comprehensive benchmark data.

Proposed method

• Integration of the SWIFT model of eye-movement control with core components of the Lewis and Vasishth sentence processing model.
• Use of Bayesian inference via Markov Chain Monte Carlo (MCMC) sampling for parameter estimation in the combined model.
• Application of profile log-likelihood analysis to identify and constrain individual model parameters in the dynamical system.
• Implementation of activation-coupled dynamics where linguistic processing influences eye movement patterns through shared neural activation states.
• Model calibration using empirical data on reading behavior, particularly interference effects from syntactic similarity.
• Use of open-source code and data repositories for reproducibility and future extension.

Experimental results

Research questions

• RQ1 Can a unified dynamical model simultaneously account for eye movement patterns and sentence processing during reading?
• RQ2 How can parameters in complex, high-dimensional dynamical systems be reliably identified and estimated?
• RQ3 To what extent can similarity-based interference in reading be reproduced through integrated linguistic and oculomotor processing?
• RQ4 What role does activation coupling play in linking sentence comprehension processes with eye movement control?
• RQ5 Is it feasible to combine established models of eye movements and sentence processing into a single, coherent computational framework?

Key findings

• SEAM successfully reproduces eye movement patterns associated with similarity-based interference in reading, such as increased fixation durations and regressions.
• The model demonstrates that integrated linguistic and oculomotor dynamics can emerge from a shared activation-based framework.
• Profile log-likelihood analysis enabled reliable identification of individual model parameters, supporting model validity and stability.
• Bayesian MCMC sampling provided robust inference for the high-dimensional parameter space of the integrated model.
• The model represents the first successful proof-of-concept integration of a complete eye-movement control model with a sentence processing model.
• All data, code, and models are publicly available via OSF and the University of Potsdam GitLab repository for reproducibility and extension.