[Paper Review] Decoding Natural Images from EEG for Object Recognition

Yonghao Song, Bingchuan Liu+4 more|arXiv (Cornell University)|Aug 25, 2023

EEG and Brain-Computer InterfacesNeuroscience8 citations

TL;DR

A self-supervised framework (NICE) decodes image representations from EEG signals via contrastive learning, enabling zero-shot object recognition on a large EEG–image dataset, with spatial attention modules to reveal brain regions involved in object recognition.

ABSTRACT

Electroencephalography (EEG) signals, known for convenient non-invasive acquisition but low signal-to-noise ratio, have recently gained substantial attention due to the potential to decode natural images. This paper presents a self-supervised framework to demonstrate the feasibility of learning image representations from EEG signals, particularly for object recognition. The framework utilizes image and EEG encoders to extract features from paired image stimuli and EEG responses. Contrastive learning aligns these two modalities by constraining their similarity. With the framework, we attain significantly above-chance results on a comprehensive EEG-image dataset, achieving a top-1 accuracy of 15.6% and a top-5 accuracy of 42.8% in challenging 200-way zero-shot tasks. Moreover, we perform extensive experiments to explore the biological plausibility by resolving the temporal, spatial, spectral, and semantic aspects of EEG signals. Besides, we introduce attention modules to capture spatial correlations, providing implicit evidence of the brain activity perceived from EEG data. These findings yield valuable insights for neural decoding and brain-computer interfaces in real-world scenarios. The code will be released on https://github.com/eeyhsong/NICE-EEG.

Motivation & Objective

• Demonstrate feasibility of learning image representations from EEG for object recognition without explicit labels.
• Build a self-supervised EEG–image alignment framework using contrastive learning.
• Investigate the temporal, spatial, spectral, and semantic aspects of EEG signals related to object recognition.
• Incorporate spatial-attention modules to reveal electrode-channel correlations.
• Provide insights into neural decoding and practical brain-computer interfaces.

Proposed method

• Use image and EEG encoders to extract modality-specific features from paired stimuli.
• Apply contrastive learning (InfoNCE) to maximize similarity of matched image–EEG pairs and minimize unmatched pairs.
• Employ a self-supervised training objective to enable zero-shot generalization to unseen image concepts.
• Adapt an EEG encoder with temporal–spatial convolutions (TSConv) and plug-in self-attention and graph-attention modules to capture spatial brain dynamics.
• Pre-train the image encoder on external image datasets (e.g., CLIP/ViT/ResNet) and align with EEG representations.
• Evaluate with 200-way zero-shot classification and analyze temporal/spatial/spectral contributions.

Experimental results

Research questions

• RQ1 Can EEG signals be used to learn image representations in a self-supervised manner for object recognition?
• RQ2 To what extent can zero-shot decoding generalize to unseen image concepts from EEG data?
• RQ3 What are the temporal, spatial, and spectral characteristics of EEG signals that support object recognition?
• RQ4 Do spatial modules (self-attention, graph attention) reveal meaningful brain region contributions to decoding?
• RQ5 How do different image encoders (ResNet, ViT, CLIP) affect EEG–image alignment and classification performance?

Key findings

• The NICE framework achieves significantly above-chance zero-shot performance on a large EEG–image dataset (top-1 13.8% in subject-dependent and 6.2% in subject-independent settings for 200-way tasks, per the reported averages).
• Incorporating self-attention or graph attention improves subject-dependent top-1 accuracy by up to ~1.8 percentage points.
• TSConv-based EEG encoding outperforms several baseline EEG encoders across subjects, with mean top-1 gains of up to 5.9 percentage points over ShallowNet and ~2.7 over Conformer.
• Pre-trained image encoders (especially CLIP) yield higher EEG–image alignment and decoding performance than non-pre-trained models.
• Temporal analysis shows a 100–600 ms window as most informative for object decoding, with occipital, temporal, and parietal regions contributing.
• Theta and beta bands (roughly 4 Hz and 14–18 Hz) show effective decoding, while gamma-band information remains elusive in this setup.