[Paper Review] DreamDiffusion: Generating High-Quality Images from Brain EEG Signals

Yunpeng Bai, Xintao Wang+4 more|arXiv (Cornell University)|Jun 29, 2023

EEG and Brain-Computer Interfaces17 citations

TL;DR

DreamDiffusion generates high-quality images directly from EEG signals by pre-training an EEG encoder with temporal masked signal modeling, and aligning EEG, text, and image spaces using CLIP supervision to condition a pre-trained Stable Diffusion model.

ABSTRACT

This paper introduces DreamDiffusion, a novel method for generating high-quality images directly from brain electroencephalogram (EEG) signals, without the need to translate thoughts into text. DreamDiffusion leverages pre-trained text-to-image models and employs temporal masked signal modeling to pre-train the EEG encoder for effective and robust EEG representations. Additionally, the method further leverages the CLIP image encoder to provide extra supervision to better align EEG, text, and image embeddings with limited EEG-image pairs. Overall, the proposed method overcomes the challenges of using EEG signals for image generation, such as noise, limited information, and individual differences, and achieves promising results. Quantitative and qualitative results demonstrate the effectiveness of the proposed method as a significant step towards portable and low-cost ``thoughts-to-image'', with potential applications in neuroscience and computer vision. The code is available here \url{https://github.com/bbaaii/DreamDiffusion}.

Motivation & Objective

• Motivate and enable image generation directly from EEG signals without translating thoughts into text.
• Leverage large-scale EEG data to learn robust semantic representations.
• Align EEG representations with text and image spaces via CLIP supervision to improve diffusion-based generation.
• Demonstrate the feasibility and potential of portable, low-cost brain-to-image generation.

Proposed method

• Temporal masked signal modeling to pre-train an EEG encoder on large-scale noisy EEG data.
• Fine-tune a pre-trained Stable Diffusion model with EEG-conditioned cross-attention using limited EEG-image pairs.
• Transform EEG embeddings to CLIP-compatible space and apply CLIP-based supervision to align EEG, text, and image embeddings.
• Use a fixed CLIP image encoder to supervise alignment between EEG-derived embeddings and image embeddings.
• Optimize SD with a loss that integrates EEG conditioning and diffusion denoising objectives.

Experimental results

Research questions

• RQ1 Can EEG signals be used to generate high-quality images without text-based translation of thoughts?
• RQ2 How can robust EEG representations be learned from noisy, variable EEG data and limited EEG-image pairs?
• RQ3 Does CLIP-based supervision improve alignment among EEG, text, and image spaces for diffusion-based generation?

Key findings

• DreamDiffusion achieves higher image quality than Brain2Image on qualitative comparisons.
• A high EEGMSM mask ratio (0.75) yields the best ablation performance on the designed tasks.
• CLIP supervision significantly improves alignment between EEG embeddings and image/text embeddings, enhancing image generation quality.
• Pre-training on ~120k EEG samples across 400+ subjects yields robust EEG representations for downstream generation.
• Fine-tuning Stable Diffusion with EEG conditioning and CLIP alignment enables realistic image synthesis from EEG signals.
• Ablation studies show the importance of pre-training, CLIP alignment, and cross-attention components for performance.