[Paper Review] EEG2TEXT: Open Vocabulary EEG-to-Text Decoding with EEG Pre-Training and Multi-View Transformer

Hanwen Liu, Daniel Hajialigol+3 more|arXiv (Cornell University)|May 3, 2024

EEG and Brain-Computer Interfaces4 citations

TL;DR

EEG2Text proposes a novel open-vocabulary EEG-to-text decoding model that leverages EEG pre-training and a multi-view transformer to improve semantic representation and decoding accuracy. By pre-training on masked raw EEG signals and modeling spatial brain regions via a multi-view transformer, it achieves state-of-the-art performance, outperforming prior methods by up to 5% in absolute BLEU and ROUGE scores.

ABSTRACT

Deciphering the intricacies of the human brain has captivated curiosity for centuries. Recent strides in Brain-Computer Interface (BCI) technology, particularly using motor imagery, have restored motor functions such as reaching, grasping, and walking in paralyzed individuals. However, unraveling natural language from brain signals remains a formidable challenge. Electroencephalography (EEG) is a non-invasive technique used to record electrical activity in the brain by placing electrodes on the scalp. Previous studies of EEG-to-text decoding have achieved high accuracy on small closed vocabularies, but still fall short of high accuracy when dealing with large open vocabularies. We propose a novel method, EEG2TEXT, to improve the accuracy of open vocabulary EEG-to-text decoding. Specifically, EEG2TEXT leverages EEG pre-training to enhance the learning of semantics from EEG signals and proposes a multi-view transformer to model the EEG signal processing by different spatial regions of the brain. Experiments show that EEG2TEXT has superior performance, outperforming the state-of-the-art baseline methods by a large margin of up to 5% in absolute BLEU and ROUGE scores. EEG2TEXT shows great potential for a high-performance open-vocabulary brain-to-text system to facilitate communication.

Motivation & Objective

• Address the challenge of accurate open-vocabulary EEG-to-text decoding, which remains limited by low performance on large, diverse vocabularies.
• Overcome the limitations of prior methods that rely on eye-tracking calibration for word-level EEG features, which introduces error and reduces generalizability.
• Improve semantic representation of raw EEG signals through end-to-end pre-training on masked raw EEG inputs.
• Enhance decoding performance by modeling brain signal processing across distinct spatial regions using a multi-view transformer architecture.
• Develop a generalizable, non-invasive brain-to-text system that supports natural language communication using only sentence-level EEG signals.

Proposed method

• Employ a base convolutional transformer model to process long, noisy raw EEG signals by extracting local and global temporal-spatial features.
• Implement a pre-training objective that masks random segments of raw EEG signals and trains the model to reconstruct the original signals, enhancing semantic learning.
• Introduce a multi-view transformer architecture where each view corresponds to a distinct spatial region of the brain, enabling region-specific signal processing.
• Utilize the pre-trained transformer as the backbone for each view in the multi-view setup, improving representation learning through shared and specialized features.
• Integrate a pre-trained BART model for autoregressive text generation, conditioned on the multi-view EEG representations.
• Train the full model end-to-end with a sequence-to-sequence objective, optimizing for BLEU and ROUGE scores on text generation.

Experimental results

Research questions

• RQ1 Can EEG pre-training on raw signals improve semantic representation learning for open-vocabulary EEG-to-text decoding?
• RQ2 How does modeling EEG signals through multiple spatial brain views enhance decoding performance compared to a single unified representation?
• RQ3 To what extent does the proposed EEG2Text model outperform existing state-of-the-art methods in open-vocabulary EEG-to-text translation?
• RQ4 Can a non-invasive EEG-based system achieve high accuracy without relying on eye-tracking for feature extraction?
• RQ5 Does the integration of pre-training and multi-view attention lead to better generalization and robustness in decoding diverse, real-world text inputs?

Key findings

• EEG2Text achieves state-of-the-art performance on open-vocabulary EEG-to-text decoding, outperforming the best baseline methods by up to 5% in absolute BLEU and ROUGE scores.
• The proposed EEG pre-training strategy significantly improves the model's ability to learn meaningful semantic representations from raw EEG signals, especially in low-resource and open-vocabulary settings.
• The multi-view transformer architecture enhances performance by capturing region-specific neural dynamics, leading to more accurate and contextually relevant text generation.
• The model demonstrates strong generalization by requiring only sentence-level EEG inputs, eliminating dependency on eye-tracking calibration and improving applicability to inner speech and natural reading scenarios.
• The ablation study confirms that both EEG pre-training and multi-view attention are critical components, with each contributing significantly to the overall performance gain.
• The model maintains high performance across diverse linguistic patterns, including homophones and semantically similar words, indicating robustness in challenging decoding scenarios.