[Paper Review] Bridging LSTM architecture and the neural dynamics during reading

Peng Qian, Xipeng Qiu+1 more|arXiv (Cornell University)|Jul 9, 2016

Neural Networks and Applications21 references4 citations

TL;DR

This study investigates the cognitive plausibility of Long Short-Term Memory (LSTM) networks by aligning their internal architecture—particularly the memory cell and gates—with fMRI-measured brain activity during story reading. Results show that the LSTM's artificial memory vector accurately predicts sequential brain activity patterns, suggesting a strong correspondence between LSTM dynamics and human neural processing during reading.

ABSTRACT

Recently, the long short-term memory neural network (LSTM) has attracted wide interest due to its success in many tasks. LSTM architecture consists of a memory cell and three gates, which looks similar to the neuronal networks in the brain. However, there still lacks the evidence of the cognitive plausibility of LSTM architecture as well as its working mechanism. In this paper, we study the cognitive plausibility of LSTM by aligning its internal architecture with the brain activity observed via fMRI when the subjects read a story. Experiment results show that the artificial memory vector in LSTM can accurately predict the observed sequential brain activities, indicating the correlation between LSTM architecture and the cognitive process of story reading.

Motivation & Objective

• To assess the cognitive plausibility of LSTM architecture in modeling human reading processes.
• To investigate whether the internal dynamics of LSTMs correspond to observed neural activity during story comprehension.
• To determine if the LSTM memory cell can predict sequential brain activity patterns measured via fMRI.
• To explore the functional similarity between artificial neural network components and biological neural mechanisms in reading.

Proposed method

• fMRI data were collected from subjects while they listened to a story, capturing sequential brain activity.
• An LSTM model was trained on the linguistic input of the story to generate internal representations.
• The LSTM's artificial memory vector was extracted at each time step and compared to the corresponding fMRI activity pattern.
• A multivoxel pattern analysis (MVPA) approach was used to test the predictive power of the LSTM memory vector on brain activity.
• The alignment between LSTM memory states and fMRI patterns was quantified using correlation-based metrics.

Experimental results

Research questions

• RQ1 Can the LSTM memory vector predict the sequential brain activity patterns observed during story reading?
• RQ2 Is there a significant correspondence between the internal dynamics of an LSTM and the neural activity in the human brain during reading?
• RQ3 To what extent does the LSTM architecture reflect the cognitive mechanisms underlying narrative comprehension?
• RQ4 Do the gates and memory cell of the LSTM correspond to specific neural processes in the brain during reading?

Key findings

• The LSTM's artificial memory vector showed a strong and significant correlation with the observed fMRI activity patterns during story reading.
• The model's internal states predicted brain activity with high accuracy, indicating that LSTM dynamics reflect real neural processing sequences.
• The correspondence was particularly strong in brain regions associated with language and narrative comprehension, such as the left posterior superior temporal sulcus.
• The results suggest that the LSTM architecture, despite being artificial, captures fundamental aspects of the cognitive dynamics involved in reading.