[Paper Review] Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity

Zijiao Chen, Jiaxin Qing+1 more|arXiv (Cornell University)|May 19, 2023

Functional Brain Connectivity StudiesNeuroscience20 citations

TL;DR

MinD-Video reconstructs high-quality, semantically meaningful videos from fMRI by decoupling an fMRI encoder from an augmented stable diffusion video generator, using progressive, multimodal, and adversarial guidance. It achieves state-of-the-art semantic accuracy and competitive pixel fidelity while offering interpretability via attention maps.

ABSTRACT

Reconstructing human vision from brain activities has been an appealing task that helps to understand our cognitive process. Even though recent research has seen great success in reconstructing static images from non-invasive brain recordings, work on recovering continuous visual experiences in the form of videos is limited. In this work, we propose Mind-Video that learns spatiotemporal information from continuous fMRI data of the cerebral cortex progressively through masked brain modeling, multimodal contrastive learning with spatiotemporal attention, and co-training with an augmented Stable Diffusion model that incorporates network temporal inflation. We show that high-quality videos of arbitrary frame rates can be reconstructed with Mind-Video using adversarial guidance. The recovered videos were evaluated with various semantic and pixel-level metrics. We achieved an average accuracy of 85% in semantic classification tasks and 0.19 in structural similarity index (SSIM), outperforming the previous state-of-the-art by 45%. We also show that our model is biologically plausible and interpretable, reflecting established physiological processes.

Motivation & Objective

• Understand how to reconstruct continuous visual experiences (videos) from non-invasive brain activity (fMRI).
• Develop a two-module pipeline separating fMRI encoding from video generation to improve quality and flexibility.
• Leverage progressive, multimodal learning and temporal attention to bridge fMRI temporal resolution gaps.
• Augment a Stable Diffusion-based video generator with scene-dynamic attention and adversarial guidance for better fidelity.

Proposed method

• Two-module pipeline: train an fMRI encoder separately from an augmented Stable Diffusion video generator, then co-train them.
• Progressive learning: large-scale MBM pre-training, followed by multimodal contrastive learning with spatiotemporal attention on windowed fMRI.
• Spatiotemporal attention to process sliding-window fMRI and account for hemodynamic delays.
• Augmented Stable Diffusion with Scene-Dynamic Sparse CA (SC) attention to condition frames on previous frames while allowing scene changes.
• Adversarial guidance via negative conditioning to enhance fMRI-conditioned sampling quality.
• Learning from brain data with interpretability: visualize attention maps to map decoder strategies onto brain networks.

Figure 1 : Brain decoding & video reconstruction . We propose a progressive learning approach to recover continuous visual experience from fMRI. High-quality videos with accurate semantics and motions are reconstructed.

Experimental results

Research questions

• RQ1 Can continuous video content be reconstructed from fMRI at arbitrary frame rates despite HRF delays?
• RQ2 Does progressive, multimodal fMRI encoding combined with a co-trained video generator improve semantic and pixel-level fidelity over prior methods?
• RQ3 How does adversarial guidance influence conditioning effectiveness and diversity of generated videos?
• RQ4 What do attention maps reveal about the brain regions and networks contributing to decoded visual content?

Key findings

• The method achieves 85% semantic classification accuracy on video content and 0.19 SSIM, outperforming prior state-of-the-art by 45%.
• The approach yields high-quality videos with accurate motion and scene dynamics across subjects.
• Attention analysis shows dominance of the visual cortex with contributions from higher cognitive networks, aligning with biological plausibility.
• Progressive learning stages reflect a shift from local to global visual features, with later layers focusing on abstract semantic information.
• Ablation studies show the importance of window size, multimodal contrastive learning, and adversarial guidance for performance.
• The framework enables reconstruction of diverse scenes and motions, including scene transitions, while maintaining frame consistency.

Figure 2 : MinD-Video Overview . Our method has two modules that are trained separately, then finetuned together. The fMRI encoder progressively learns fMRI features through multiple stages, including SC-MBM pre-training and multimodal contrastive learning. A spatiotemporal attention is designed to