[Paper Review] Mind Reader: Reconstructing complex images from brain activities

Sikun Lin, Thomas C. Sprague+1 more|arXiv (Cornell University)|Sep 30, 2022

Cell Image Analysis Techniques34 citations

TL;DR

The paper reconstructs complex, semantically rich images from fMRI by mapping brain signals into a pre-aligned vision-language latent space (CLIP) and using a conditional generator to render images conditioned on that space and image captions.

ABSTRACT

Understanding how the brain encodes external stimuli and how these stimuli can be decoded from the measured brain activities are long-standing and challenging questions in neuroscience. In this paper, we focus on reconstructing the complex image stimuli from fMRI (functional magnetic resonance imaging) signals. Unlike previous works that reconstruct images with single objects or simple shapes, our work aims to reconstruct image stimuli that are rich in semantics, closer to everyday scenes, and can reveal more perspectives. However, data scarcity of fMRI datasets is the main obstacle to applying state-of-the-art deep learning models to this problem. We find that incorporating an additional text modality is beneficial for the reconstruction problem compared to directly translating brain signals to images. Therefore, the modalities involved in our method are: (i) voxel-level fMRI signals, (ii) observed images that trigger the brain signals, and (iii) textual description of the images. To further address data scarcity, we leverage an aligned vision-language latent space pre-trained on massive datasets. Instead of training models from scratch to find a latent space shared by the three modalities, we encode fMRI signals into this pre-aligned latent space. Then, conditioned on embeddings in this space, we reconstruct images with a generative model. The reconstructed images from our pipeline balance both naturalness and fidelity: they are photo-realistic and capture the ground truth image contents well.

Motivation & Objective

• Investigate decoding of complex, real-world scene images from fMRI signals.
• Assess whether adding a text modality improves image reconstruction over visual-only mappings.
• Leverage a large, aligned vision-language latent space to mitigate fMRI data scarcity.
• Demonstrate a two-stage training approach that separates brain-to-space mapping from space-to-image generation.

Proposed method

• Map ROI-reduced fMRI signals to CLIP image embeddings and CLIP caption embeddings using CNN-based mappers.
• Screen image captions with CLIP encoders to select high-quality captions per image (caption screening).
• Use two separate mapping models (fMRI to h_img and fMRI to h_cap) to align with image and caption CLIP embeddings.
• Fine-tune a conditional generator (Lafite-style) on CLIP-aligned embeddings to produce images conditioned on both h_img' and h_cap' embeddings.
• Apply multi-component losses (MSE, cosine similarity, contrastive losses) and a three-head discriminator to enforce realism and cross-modal alignment.
• Train in two stages: (i) learn fMRI→CLIP embeddings; (ii) finetune the conditional generator with frozen mappers.

Experimental results

Research questions

• RQ1 Can fMRI signals be mapped into a shared CLIP space that jointly encodes image content and textual captions?
• RQ2 Does incorporating a text modality (captions) improve reconstruction fidelity for complex scenes compared to image-only mappings?
• RQ3 How does a pre-trained, multimodal generator perform when conditioned on brain-derived CLIP embeddings?
• RQ4 What is the impact of training strategy (frozen mappers vs end-to-end) on reconstruction quality?

Key findings

• The pipeline produces photo-realistic reconstructions that preserve content and relationships in complex scenes.
• Mapping fMRI to CLIP embeddings (especially via the image path) yields better FID performance than caption-only mappings in most settings.
• Using a two-head conditioning (image and caption CLIP embeddings) with a LF-Lafite pre-trained generator outperforms single-head setups under several metrics.
• End-to-end finetuning of mappers with GAN objectives generally degrades performance and can collapse embedding representations.
• CLIP space serves as an effective intermediary for brain decoding, with fMRI and CLIP embeddings providing complementary information for reconstruction.