[Paper Review] Understanding language-elicited EEG data by predicting it from a fine-tuned language model

Dan Schwartz, Tom M. Mitchell|arXiv (Cornell University)|Apr 2, 2019

Neurobiology of Language and Bilingualism30 references4 citations

TL;DR

This paper proposes a novel method to understand language-elicited EEG responses by fine-tuning a pre-trained language model to predict event-related potentials (ERPs) from brain activity. It demonstrates, for the first time, that all six major ERP components—previously thought to be predictable only in part—can be predicted from language model embeddings, revealing new relationships through joint multitask training, especially between syntactic (LAN/P600, ELAN/P600) and semantic (N400/EPNP/PNP) components.

ABSTRACT

Electroencephalography (EEG) recordings of brain activity taken while participants read or listen to language are widely used within the cognitive neuroscience and psycholinguistics communities as a tool to study language comprehension. Several time-locked stereotyped EEG responses to word-presentations -- known collectively as event-related potentials (ERPs) -- are thought to be markers for semantic or syntactic processes that take place during comprehension. However, the characterization of each individual ERP in terms of what features of a stream of language trigger the response remains controversial. Improving this characterization would make ERPs a more useful tool for studying language comprehension. We take a step towards better understanding the ERPs by fine-tuning a language model to predict them. This new approach to analysis shows for the first time that all of the ERPs are predictable from embeddings of a stream of language. Prior work has only found two of the ERPs to be predictable. In addition to this analysis, we examine which ERPs benefit from sharing parameters during joint training. We find that two pairs of ERPs previously identified in the literature as being related to each other benefit from joint training, while several other pairs of ERPs that benefit from joint training are suggestive of potential relationships. Extensions of this analysis that further examine what kinds of information in the model embeddings relate to each ERP have the potential to elucidate the processes involved in human language comprehension.

Motivation & Objective

• To improve characterization of event-related potentials (ERPs) in language comprehension by linking them to features of language input.
• To investigate whether language model embeddings can predict neural responses (ERPs) elicited by spoken or written language.
• To explore how joint training across multiple ERP components reveals underlying cognitive relationships.
• To determine whether bidirectional language models outperform unidirectional ones in predicting ERP responses.
• To examine whether behavioral data (eye-tracking, self-paced reading) can be jointly learned with ERP data to improve prediction.

Proposed method

• Fine-tune a pre-trained bidirectional transformer language model (e.g., BERT) to predict six ERP components from word-level language inputs.
• Use a multitask learning framework where the model predicts multiple ERP components simultaneously, sharing representations across tasks.
• Train the model on a dataset of word-by-word language stimuli paired with corresponding ERP amplitudes from Frank et al. (2015).
• Compare performance of bidirectional versus unidirectional language model variants in predicting ERP components.
• Integrate behavioral data (eye-tracking, self-paced reading) as auxiliary signals in the multitask learning setup to improve ERP prediction.
• Analyze learned representations to identify which linguistic features in the model embeddings correlate with each ERP component.

Experimental results

Research questions

• RQ1 Can a fine-tuned language model predict all six major ERP components from language input, despite prior work only showing predictability for two?
• RQ2 Which ERP components benefit from joint training, and what does this imply about their underlying cognitive relationships?
• RQ3 Why does a bidirectional language model outperform a unidirectional one in predicting ERP responses?
• RQ4 Can behavioral data (e.g., eye-tracking, reading time) improve ERP prediction when used in a multitask learning setup?
• RQ5 What linguistic features in the model's hidden representations are most predictive of each ERP component?

Key findings

• All six ERP components—N400, EPNP, PNP, P600, LAN, and ELAN—can be predicted from language model embeddings, extending prior findings that only two were predictable.
• Joint training improves prediction performance for specific ERP pairs, including LAN+P600 and ELAN+P600, which were previously identified as cognitively related.
• The bidirectional model outperforms the unidirectional model in predicting ERP responses, suggesting that future context aids in modeling brain activity.
• Multitask learning with behavioral data (eye-tracking, self-paced reading) improves ERP prediction, indicating shared representations across neural and behavioral signals.
• The model reveals previously unknown ERP relationships, such as EPNP and PNP benefiting from joint training, suggesting potential functional links.
• The results suggest that ERP components are not isolated markers but are jointly shaped by shared linguistic and cognitive processes, as revealed through joint representation learning.