[Paper Review] DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation

Yiqun Duan, Jinzhao Zhou+3 more|arXiv (Cornell University)|Sep 25, 2023

EEG and Brain-Computer Interfaces15 citations

TL;DR

DeWave introduces discrete codex encoding to translate both word-level EEG features and raw EEG waves into open-vocabulary text, achieving state-of-the-art BLEU/ROUGE on ZuCo and enabling raw-wave translation without event markers.

ABSTRACT

The translation of brain dynamics into natural language is pivotal for brain-computer interfaces (BCIs). With the swift advancement of large language models, such as ChatGPT, the need to bridge the gap between the brain and languages becomes increasingly pressing. Current methods, however, require eye-tracking fixations or event markers to segment brain dynamics into word-level features, which can restrict the practical application of these systems. To tackle these issues, we introduce a novel framework, DeWave, that integrates discrete encoding sequences into open-vocabulary EEG-to-text translation tasks. DeWave uses a quantized variational encoder to derive discrete codex encoding and align it with pre-trained language models. This discrete codex representation brings forth two advantages: 1) it realizes translation on raw waves without marker by introducing text-EEG contrastive alignment training, and 2) it alleviates the interference caused by individual differences in EEG waves through an invariant discrete codex with or without markers. Our model surpasses the previous baseline (40.1 and 31.7) by 3.06% and 6.34%, respectively, achieving 41.35 BLEU-1 and 33.71 Rouge-F on the ZuCo Dataset. This work is the first to facilitate the translation of entire EEG signal periods without word-level order markers (e.g., eye fixations), scoring 20.5 BLEU-1 and 29.5 Rouge-1 on the ZuCo Dataset.

Motivation & Objective

• Bridge brain dynamics and natural language by enabling open-vocabulary EEG-to-text translation without relying on eye-tracking markers or event tags.
• Introduce a discrete codex representation that aligns EEG features with a pre-trained language model.
• Enable translation directly from raw EEG waves through self-supervised encoding and text-aligned contrastive learning.

Proposed method

• Use a vector-quantized variational encoder to produce a discrete codex from EEG embeddings.
• Align the discrete codex with a pre-trained language model (BART) for generation of text.
• Train with a combination of self-reconstruction loss, contrastive EEG-text alignment, and codex consistency loss.
• For word-level EEG, encode eye-fixation-segmented features into embeddings and map to codex tokens.
• For raw EEG waves, employ a self-supervised Conformer-based encoder to produce embeddings that are quantized to codex tokens.

Experimental results

Research questions

• RQ1 Can discrete codex encoding improve robustness to inter-subject EEG variability for open-vocabulary EEG-to-text translation?
• RQ2 Is it feasible to translate EEG signals into text without word-level event markers (e.g., eye fixations) using a discrete latent space?
• RQ3 How does aligning EEG-derived codex tokens with a pre-trained language model affect translation quality for EEG-to-text tasks?

Key findings

• On word-level EEG features, DeWave achieves BLEU-1 of 41.35 and ROUGE-1 of 28.82, outperforming the baseline by 1.23 BLEU-1 and 0.00 ROUGE-1 in one metric setting.
• On word-level EEG features, DeWave reaches BLEU-1 41.35, BLEU-2 24.15, BLEU-3 13.92, BLEU-4 8.22, ROUGE-1 recall 28.82, ROUGE-1 precision 33.71, ROUGE-1 F1 30.69.
• On raw EEG waves, DeWave achieves BLEU-1 20.51 and ROUGE-1 21.18, significantly surpassing baselines that use simple time-window slicing.
• With contrastive learning (DeWave+Contrastive), BLEU-1 improves to 21.09 and ROUGE-1 to 22.01, indicating joint benefits from cross-modal alignment.
• Compared to prior EEG-to-text baselines on ZuCo, DeWave%improves BLEU-1 by about 3.06% and ROUGE-1 by about 6.34% in the open-vocabulary setting (word-level features).
• DeWave demonstrates the first raw EEG-wave to text translation without markers, achieving BLEU-1 20.51 and ROUGE-1 29.42 under the open vocabulary framework.