[Paper Review] Decoding natural image stimuli from fMRI data with a surface-based convolutional network

Zijin Gu, K. A. Jamison+2 more|arXiv (Cornell University)|Dec 5, 2022

CCD and CMOS Imaging Sensors13 citations

TL;DR

This paper proposes Cortex2Image, a novel surface-based convolutional network that decodes natural image stimuli from fMRI data by leveraging cortical topology and a variational training approach with an Instance-Conditioned GAN. It achieves state-of-the-art semantic fidelity and improved fine-grained detail reconstruction by modeling spatial relationships in brain activity across the entire cortex, outperforming prior methods in both qualitative and quantitative metrics.

ABSTRACT

Due to the low signal-to-noise ratio and limited resolution of functional MRI data, and the high complexity of natural images, reconstructing a visual stimulus from human brain fMRI measurements is a challenging task. In this work, we propose a novel approach for this task, which we call Cortex2Image, to decode visual stimuli with high semantic fidelity and rich fine-grained detail. In particular, we train a surface-based convolutional network model that maps from brain response to semantic image features first (Cortex2Semantic). We then combine this model with a high-quality image generator (Instance-Conditioned GAN) to train another mapping from brain response to fine-grained image features using a variational approach (Cortex2Detail). Image reconstructions obtained by our proposed method achieve state-of-the-art semantic fidelity, while yielding good fine-grained similarity with the ground-truth stimulus. Our code is available at: https://github.com/zijin-gu/meshconv-decoding.git.

Motivation & Objective

• To address the limitations of prior fMRI-based image decoding methods that rely on region-of-interest (ROI) selection and vectorized voxel responses, which ignore cortical spatial topology and introduce subjectivity.
• To improve image reconstruction quality by modeling the 2D spatial structure of cortical surface activity using surface-based convolutions.
• To enhance computational efficiency and reconstruction fidelity by training a joint Cortex2Detail model end-to-end with a pre-trained image generator (IC-GAN) via a variational approach.
• To achieve high semantic fidelity and rich fine-grained detail in reconstructed images from whole-cortex fMRI responses, surpassing existing state-of-the-art methods.
• To demonstrate that subject-specific models do not generalize well across individuals, highlighting the need for individualized yet shared architectural frameworks.

Proposed method

• The method employs a surface-based convolutional neural network (Cortex2Semantic) that maps whole-cortex fMRI responses to semantic image features, preserving spatial topology via standardized cortical mesh representations.
• A second model, Cortex2Detail, is trained end-to-end with a pre-trained Instance-Conditioned GAN (IC-GAN) using a variational inference framework to generate high-fidelity image details from brain responses.
• The IC-GAN is frozen during training, and the detail vector is inferred via a variational autoencoder-like optimization, reducing the need for iterative noise vector optimization used in prior work.
• The framework uses the Natural Scenes Dataset (NSD), with 8 subjects scanned at 7T, viewing 9,000–10,000 natural images, to train and evaluate the models.
• Brain responses are processed using a standardized cortical surface mesh, avoiding ROI-dependent preprocessing and enabling a shared architecture across subjects.
• The method jointly optimizes semantic and detail reconstruction by combining feature-level supervision with adversarial training on image-level realism.

Experimental results

Research questions

• RQ1 Can a surface-based convolutional network that respects the 2D topology of the cortical surface improve semantic and fine-grained image reconstruction from fMRI compared to vectorized voxel approaches?
• RQ2 Does end-to-end training with a pre-trained GAN reduce computational cost and improve reconstruction quality compared to iterative optimization of noise vectors?
• RQ3 To what extent does modeling whole-cortex activity, rather than restricted ROIs, enhance decoding performance and generalization across subjects?
• RQ4 How do the proposed models compare to state-of-the-art methods in terms of semantic accuracy and perceptual similarity to ground-truth stimuli?
• RQ5 Why do subject-specific models fail to generalize, and how does this affect the feasibility of cross-subject decoding in neural decoding tasks?

Key findings

• The Cortex2Image model achieves state-of-the-art semantic fidelity in image reconstruction, correctly identifying object categories such as zebras, trains, and airplanes in well-decoded examples.
• The model preserves fine-grained details such as zebra stripes, color of food, object orientation, and spatial layout (e.g., train on track, plane in sky), which are often lost in prior methods.
• Compared to the RidgeImage baseline, Cortex2Image outperforms in all evaluation metrics except SSIM, with significant gains in FID, LPIPS, and CLIP scores, indicating better perceptual and semantic quality.
• The Cortex2Semantic model alone captures object categories but fails to reconstruct fine details; the full Cortex2Image model significantly improves detail fidelity, demonstrating the value of joint semantic and detail learning.
• The RidgeImage baseline produces poor visual quality despite some semantic accuracy, highlighting the limitations of ridge regression with hand-crafted features and non-spatial modeling.
• Subject-specific models do not generalize across individuals, and performance varies significantly, likely due to differences in signal-to-noise ratio and neural response properties across subjects.