[Paper Review] ViT2EEG: Leveraging Hybrid Pretrained Vision Transformers for EEG Data

Ruiqi Yang, Eric Modesitt|arXiv (Cornell University)|Aug 1, 2023

EEG and Brain-Computer Interfaces7 citations

TL;DR

This paper proposes fine-tuning a hybrid Vision Transformer (ViT) pretrained on ImageNet for EEG regression, demonstrating that pretraining on unrelated image data significantly boosts performance on EEG tasks. The method outperforms both standard CNNs and non-pretrained ViTs, showing that transfer learning from vision to EEG is highly effective despite domain differences.

ABSTRACT

In this study, we demonstrate the application of a hybrid Vision Transformer (ViT) model, pretrained on ImageNet, on an electroencephalogram (EEG) regression task. Despite being originally trained for image classification tasks, when fine-tuned on EEG data, this model shows a notable increase in performance compared to other models, including an identical architecture ViT trained without the ImageNet weights. This discovery challenges the traditional understanding of model generalization, suggesting that Transformer models pretrained on seemingly unrelated image data can provide valuable priors for EEG regression tasks with an appropriate fine-tuning pipeline. The success of this approach suggests that the features extracted by ViT models in the context of visual tasks can be readily transformed for the purpose of EEG predictive modeling. We recommend utilizing this methodology not only in neuroscience and related fields, but generally for any task where data collection is limited by practical, financial, or ethical constraints. Our results illuminate the potential of pretrained models on tasks that are clearly distinct from their original purpose.

Motivation & Objective

• To investigate whether Vision Transformers pretrained on ImageNet can be effectively fine-tuned for EEG regression tasks.
• To evaluate the performance of a hybrid ViT model against conventional CNNs and non-pretrained ViTs on EEG data.
• To assess the impact of pretraining on model generalization and performance in low-data regimes common in EEG research.
• To challenge the assumption that models must be pre-trained on similar data types for effective transfer learning.
• To provide a scalable, data-efficient approach for EEG analysis that reduces reliance on costly data collection.

Proposed method

• Fine-tuning a hybrid Vision Transformer (ViT) pretrained on ImageNet for EEG regression using the EEGEyeNet dataset.
• Using a hybrid architecture that combines convolutional layers for initial feature extraction with a Vision Transformer for global attention and sequence modeling.
• Applying standard self-attention mechanisms in the ViT blocks to capture long-range dependencies in spatio-temporal EEG signals.
• Training the model end-to-end on EEG data with a regression head to predict gaze positions from EEG inputs.
• Employing standard optimization and regularization techniques, including weight decay and dropout, to stabilize training.
• Comparing performance against baseline models such as EEGNet, Xception, and a non-pretrained ViT-Base.

Figure 1. EEGEyeNet Large Grid Paradigm (Kastrati et al . , 2021 ) . Participants are asked to fixate on particular dots in a given period.

Experimental results

Research questions

• RQ1 Can a Vision Transformer pretrained on ImageNet be effectively fine-tuned for EEG regression without architectural modification?
• RQ2 Does pretraining on ImageNet provide a performance advantage over training a ViT from random initialization on EEG data?
• RQ3 How does the performance of a pretrained ViT compare to state-of-the-art CNN-based models on EEGEyeNet?
• RQ4 To what extent does the inductive bias of CNNs limit performance on EEG regression compared to attention-based models?
• RQ5 Can transfer learning from vision data serve as a viable solution to data scarcity in EEG research?

Key findings

• The pretrained ViT model achieved a Root Mean Squared Error (RMSE) of 55.4 mm on the EEGEyeNet gaze prediction task, significantly outperforming the previous state-of-the-art CNN model, which had an RMSE of 70.4 mm.
• The pretrained ViT model also outperformed a non-pretrained ViT-Base, demonstrating that pretraining contributes meaningfully to performance beyond architecture alone.
• The model's superior performance is primarily attributed to the Transformer block's ability to capture long-range dependencies, not just the convolutional front-end.
• The model's performance surpasses even baseline models like EEGNet and Xception, which use depthwise convolutions, indicating the advantage of self-attention over local convolutions in EEG data.
• The results suggest that pretraining on large-scale image data provides strong inductive bias that generalizes well to EEG regression, despite domain shift.
• The study demonstrates that transfer learning from vision to EEG is feasible and effective, even when the source and target tasks are unrelated.

Figure 2. Proposed EEGViT, a hybrid ViT (Vision Transformer) architecture designed specifically for EEG raw signal as input. A two-step convolution operation is applied to generate patch embeddings. Then we add positional embeddings and pass the resulting sequence into ViT layers. The illustration o