[Paper Review] MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion

Yizhuo Lu, Changde Du+2 more|arXiv (Cornell University)|Mar 24, 2023

Cell Image Analysis TechniquesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

MindDiffuser is a two-stage diffusion model that reconstructs semantically accurate and structurally aligned images from fMRI data by combining CLIP text embeddings, VQ-VAE latents, and CLIP visual features. It achieves state-of-the-art performance on the Natural Scenes Dataset (NSD) by jointly optimizing semantic and structural fidelity through iterative refinement with backpropagation.

ABSTRACT

Reconstructing visual stimuli from measured functional magnetic resonance imaging (fMRI) has been a meaningful and challenging task. Previous studies have successfully achieved reconstructions with structures similar to the original images, such as the outlines and size of some natural images. However, these reconstructions lack explicit semantic information and are difficult to discern. In recent years, many studies have utilized multi-modal pre-trained models with stronger generative capabilities to reconstruct images that are semantically similar to the original ones. However, these images have uncontrollable structural information such as position and orientation. To address both of the aforementioned issues simultaneously, we propose a two-stage image reconstruction model called MindDiffuser, utilizing Stable Diffusion. In Stage 1, the VQ-VAE latent representations and the CLIP text embeddings decoded from fMRI are put into the image-to-image process of Stable Diffusion, which yields a preliminary image that contains semantic and structural information. In Stage 2, we utilize the low-level CLIP visual features decoded from fMRI as supervisory information, and continually adjust the two features in Stage 1 through backpropagation to align the structural information. The results of both qualitative and quantitative analyses demonstrate that our proposed model has surpassed the current state-of-the-art models in terms of reconstruction results on Natural Scenes Dataset (NSD). Furthermore, the results of ablation experiments indicate that each component of our model is effective for image reconstruction.

Motivation & Objective

• Address the dual challenge of generating semantically accurate and structurally precise image reconstructions from fMRI signals.
• Overcome limitations of prior methods: optimized models lack semantic richness, while generative models lack structural control.
• Enable robust, subject-agnostic image reconstruction despite inter-subject variability in fMRI responses.
• Leverage multimodal pre-trained models (CLIP, Stable Diffusion) to improve reconstruction quality and controllability.

Proposed method

• Stage 1 decodes fMRI into CLIP text embeddings (c) and VQ-VAE latent codes (z), then injects noise into z to form z_T for diffusion.
• Uses cross-attention in Stable Diffusion's image-to-image denoising to fuse c and z_T, generating an initial image with semantic and detail information.
• Stage 2 employs CLIP’s low-level visual features (decoded from fMRI) as supervisory signals to guide backpropagation.
• Continuously refines the latent features (c and z) through backpropagation to align the reconstructed image’s structure with the original.
• Utilizes a two-stage training-free inference pipeline: first generate a semantically plausible image, then optimize its structure via gradient-based refinement.
• Employs VQ-VAE for discrete latent representation and Stable Diffusion for high-fidelity image generation with conditional control.

Experimental results

Research questions

• RQ1 Can a diffusion-based model jointly achieve high semantic and structural fidelity in fMRI-to-image reconstruction?
• RQ2 How effective is the integration of CLIP text and visual features in guiding image generation from fMRI?
• RQ3 To what extent does fine-grained latent detail (z) influence optimization stability and reconstruction quality?
• RQ4 Can the model generalize across subjects without retraining or fine-tuning, despite inter-subject fMRI variability?
• RQ5 How does the two-stage refinement process compare to end-to-end optimization or random z initialization in terms of reconstruction quality?

Key findings

• MindDiffuser outperforms state-of-the-art models in both qualitative and quantitative evaluations on the Natural Scenes Dataset (NSD).
• Ablation studies confirm that both CLIP text features (c) and decoded low-level visual features are essential for high-quality reconstruction.
• Random initialization of the VQ-VAE latent code z leads to suboptimal convergence, while using decoded z enables stable, accurate optimization.
• The model achieves semantically and structurally aligned reconstructions for subjects 1, 2, 5, and 7 without any subject-specific adaptation.
• Reconstructions of complex stimuli like 'airplane at sunset' and 'bathroom' show high fidelity in shape, orientation, and semantic content.
• The use of CLIP visual features as supervisory signals significantly improves structural alignment, especially in early optimization steps.