[Paper Review] Accurate reconstruction of image stimuli from human fMRI based on the decoding model with capsule network architecture

Kai Qiao, Chi Zhang+5 more|arXiv (Cornell University)|Jan 2, 2018

EEG and Brain-Computer Interfaces25 references18 citations

TL;DR

This paper proposes a novel decoding model, CNAVR, that reconstructs image stimuli from human fMRI data using a capsule network (CapsNet) architecture to improve feature representation and structural similarity. By learning end-to-end nonlinear mappings from fMRI to high-level capsule features and back to images, the method achieves a 10% improvement in SSIM over state-of-the-art methods on handwritten digit reconstruction.

ABSTRACT

In neuroscience, all kinds of computation models were designed to answer the open question of how sensory stimuli are encoded by neurons and conversely, how sensory stimuli can be decoded from neuronal activities. Especially, functional Magnetic Resonance Imaging (fMRI) studies have made many great achievements with the rapid development of the deep network computation. However, comparing with the goal of decoding orientation, position and object category from activities in visual cortex, accurate reconstruction of image stimuli from human fMRI is a still challenging work. In this paper, the capsule network (CapsNet) architecture based visual reconstruction (CNAVR) method is developed to reconstruct image stimuli. The capsule means containing a group of neurons to perform the better organization of feature structure and representation, inspired by the structure of cortical mini column including several hundred neurons in primates. The high-level capsule features in the CapsNet includes diverse features of image stimuli such as semantic class, orientation, location and so on. We used these features to bridge between human fMRI and image stimuli. We firstly employed the CapsNet to train the nonlinear mapping from image stimuli to high-level capsule features, and from high-level capsule features to image stimuli again in an end-to-end manner. After estimating the serviceability of each voxel by encoding performance to accomplish the selecting of voxels, we secondly trained the nonlinear mapping from dimension-decreasing fMRI data to high-level capsule features. Finally, we can predict the high-level capsule features with fMRI data, and reconstruct image stimuli with the CapsNet. We evaluated the proposed CNAVR method on the dataset of handwritten digital images, and exceeded about 10% than the accuracy of all existing state-of-the-art methods on the structural similarity index (SSIM).

Motivation & Objective

• To address the challenge of accurately reconstructing visual stimuli from human fMRI data, which remains difficult despite advances in deep learning.
• To improve the representation of spatial and semantic features (e.g., orientation, location, class) in fMRI decoding by leveraging capsule networks.
• To develop an end-to-end learning framework that maps fMRI activity to high-level capsule features and reconstructs images from them.
• To identify and utilize the most informative voxels in fMRI data based on their encoding performance for improved reconstruction fidelity.
• To bridge the gap between neural activity in the visual cortex and perceptual image reconstruction using a hierarchical, attention-like feature organization.

Proposed method

• The CapsNet architecture is used to learn a nonlinear mapping from image stimuli to high-level capsule features, which encode diverse attributes such as orientation, location, and semantic class.
• A two-stage training process is employed: first, the CapsNet is trained to map images to capsule features and back to images in an end-to-end manner.
• Voxel selection is performed by evaluating each voxel’s encoding performance, retaining only the most informative voxels to reduce dimensionality and improve signal relevance.
• A second nonlinear mapping is trained from dimension-reduced fMRI data to high-level capsule features, enabling prediction of these features from brain activity.
• Image reconstruction is achieved by decoding the predicted capsule features back into pixel-level images using the trained CapsNet.
• The model is evaluated using a dataset of handwritten digit images, with performance measured via structural similarity index (SSIM).

Experimental results

Research questions

• RQ1 Can capsule networks improve the accuracy of image reconstruction from human fMRI data compared to traditional deep learning models?
• RQ2 How effectively can capsule networks represent hierarchical visual features such as orientation, location, and semantic class in fMRI decoding?
• RQ3 Does end-to-end training of a CapsNet-based decoder yield better reconstruction fidelity than conventional autoencoder or fully connected architectures?
• RQ4 To what extent do voxel selection based on encoding performance enhance reconstruction quality in fMRI decoding?
• RQ5 Can the attention-like routing mechanism in capsule networks better capture spatial relationships in reconstructed images than standard pooling or convolutional layers?

Key findings

• The proposed CNAVR method achieved a 10% improvement in structural similarity index (SSIM) over existing state-of-the-art methods in reconstructing handwritten digit images from fMRI data.
• The capsule network's ability to encode spatial hierarchies and pose-related features significantly enhanced the quality of reconstructed images.
• Voxel selection based on encoding performance effectively reduced noise and irrelevant signals, improving the robustness of the decoding process.
• The end-to-end training of the CapsNet enabled better generalization and more accurate reconstruction compared to models with separate feature extraction and decoding stages.
• The high-level capsule features successfully captured semantic, spatial, and structural attributes of the stimuli, enabling faithful image reconstruction from fMRI patterns.
• The method demonstrated superior performance in reconstructing fine-grained visual details and object structures, indicating the effectiveness of capsule-based representation in brain decoding.