[Paper Review] DREAM: Visual Decoding from Reversing Human Visual System

Weihao Xia, Raoul de Charette+2 more|arXiv (Cornell University)|Oct 3, 2023

Visual Attention and Saliency Detection52 references4 citations

TL;DR

DREAM proposes a novel fMRI-to-image reconstruction method that reverses the human visual system's hierarchical and parallel pathways to decode semantics, color, and depth from brain activity. By introducing Reverse-VAC for semantics and Reverse-PKM for joint color-depth prediction, it outperforms state-of-the-art methods in appearance, structure, and semantic consistency, achieving superior reconstruction fidelity on the NSD dataset.

ABSTRACT

In this work we present DREAM, an fMRI-to-image method for reconstructing viewed images from brain activities, grounded on fundamental knowledge of the human visual system. We craft reverse pathways that emulate the hierarchical and parallel nature of how humans perceive the visual world. These tailored pathways are specialized to decipher semantics, color, and depth cues from fMRI data, mirroring the forward pathways from visual stimuli to fMRI recordings. To do so, two components mimic the inverse processes within the human visual system: the Reverse Visual Association Cortex (R-VAC) which reverses pathways of this brain region, extracting semantics from fMRI data; the Reverse Parallel PKM (R-PKM) component simultaneously predicting color and depth from fMRI signals. The experiments indicate that our method outperforms the current state-of-the-art models in terms of the consistency of appearance, structure, and semantics. Code will be made publicly available to facilitate further research in this field.

Motivation & Objective

• To address the limitations of current fMRI-to-image decoding methods that suffer from poor appearance and structural consistency due to reliance on vision-language models like CLIP.
• To incorporate biologically plausible mechanisms from the human visual system (HVS) into visual decoding, particularly for color, depth, and semantics.
• To reduce information loss during fMRI-to-image translation by emulating the forward HVS pathway in reverse, using specialized neural components.
• To improve image reconstruction quality by integrating color and depth cues derived directly from fMRI signals, rather than relying on indirect supervision.
• To demonstrate that biologically grounded reverse pathways yield more consistent and perceptually accurate reconstructions than existing diffusion-based approaches.

Proposed method

• DREAM employs a reverse pathway inspired by the human visual system, modeling the inverse of the forward HVS process from visual stimuli to fMRI.
• The Reverse Visual Association Cortex (R-VAC) component decodes high-level semantics from fMRI signals, producing CLIP-style embeddings to guide image generation.
• The Reverse Parallel PKM (R-PKM) component simultaneously predicts color (as spatial palettes) and depth (as depth maps) from fMRI data, providing structural and chromatic guidance.
• Deciphered cues—semantics, color, and depth—are fed into Stable Diffusion with T2I-Adapter to reconstruct images, using the Color Adapter and Depth Adapter for spatial control.
• The method is trained subject-specifically on the NSD dataset, with individual models retrained per participant to account for inter-subject variability in fMRI patterns.
• A ×64 downsampling strategy is applied to initial guessed images to extract robust color palettes while mitigating noise from low-fidelity initial reconstructions.

Experimental results

Research questions

• RQ1 Can reverse modeling of the human visual system’s hierarchical and parallel pathways improve fMRI-to-image reconstruction quality?
• RQ2 Does explicitly decoding color and depth cues from fMRI signals lead to more consistent image reconstructions than relying solely on CLIP-aligned semantics?
• RQ3 Can biologically inspired reverse pathways outperform existing diffusion-based methods that depend on vision-language alignment?
• RQ4 To what extent do subject-specific fMRI patterns affect reconstruction performance, and can consistent errors reveal dataset biases?
• RQ5 How do color and depth predictions from fMRI compare to ground truth, and can they serve as effective guidance for image generation?

Key findings

• DREAM achieves state-of-the-art performance on the NSD dataset, with subject-specific reconstructions showing high consistency in appearance, structure, and semantics.
• Quantitative metrics on four subjects (sub01–sub07) show PixCorr values between 0.265 and 0.288, SSIM between 92.7% and 97.5%, and CLIP scores ranging from 93.7% to 95.2%.
• The method demonstrates stable performance across all participants, with low variance in metrics, indicating robustness to inter-subject fMRI variability.
• Color and depth predictions from the R-PKM component, though coarse, provide meaningful spatial guidance, improving reconstruction fidelity despite resolution limitations.
• Shared reconstruction errors—such as misclassifying vase-flower images as paintings or angled food plates as top-down views—suggest dataset biases that persist across subjects.
• Visual results show that incorporating color and depth cues significantly reduces appearance inconsistencies seen in prior methods, such as those in [31], where semantics were preserved but colors were distorted.