[Paper Review] From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI

Roman Beliy, Guy Gaziv+4 more|arXiv (Cornell University)|Jul 3, 2019

Advanced MRI Techniques and Applications31 references58 citations

TL;DR

The paper introduces a self-supervised framework that enables fMRI-to-image reconstruction using both labeled pairs and unlabeled data, including unlabeled test-fMRI, to adapt to new test statistics and improve reconstruction quality.

ABSTRACT

Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training with both types of unlabeled data. Importantly, it allows training on the unlabeled test-fMRI data. This self-supervision adapts the reconstruction network to the new input test-data, despite its deviation from the statistics of the scarce training data.

Motivation & Objective

• Address the scarcity of labeled Image–fMRI pairs for reconstructing seen images from fMRI.
• Leverage unlabeled data (images and fMRI) to improve decoding performance.
• Adapt the reconstruction model to test-time data statistics by training on unlabeled test-fMRI.
• Develop a two-phase training pipeline that maintains encoder stability while enhancing decoder capabilities.
• Demonstrate robustness and generalization across two public fMRI datasets.

Proposed method

• Train an Encoder that maps images to fMRI responses using supervised image–fMRI pairs.
• Train a Decoder that maps fMRI to images, using a combined loss with labeled data and self-supervised losses.
• Incorporate unsupervised training on unlabeled images via the Encoder–Decoder path (s -> D(E(s)) to enforce image consistency).
• Incorporate unsupervised training on unlabeled test-fMRI via the Decoder–Encoder path (r -> E(D(r)) to adapt to test statistics).
• Keep the Encoder fixed during Decoder training to preserve fMRI prediction consistency with unsupervised objectives.
• Use a three-part Decoder loss L^D + L^ED + L^DE combining supervised image reconstruction loss with features and color losses and total variation regularization.

Experimental results

Research questions

• RQ1 Can self-supervision on unlabeled data (images and test-fMRI) improve fMRI-to-image reconstruction beyond supervised learning?
• RQ2 How does adapting to unlabeled test-fMRI influence reconstruction quality of the target test image?
• RQ3 Does training with external unlabeled images help, and is adapting to target test-fMRI more impactful?
• RQ4 What is the impact of a two-phase training approach on stability and reconstruction fidelity?

Key findings

• Self-supervision on unlabeled test-fMRI provides the greatest boost in reconstruction quality.
• Unsupervised training on unlabeled natural images offers some improvement but is less impactful than test-fMRI adaptation.
• The full method with both unlabeled data types achieves higher identification rates than supervised training alone.
• In 2-way identification, supervised alone reached 80.1% which rose to 85.3% with full unlabeled-data training.
• For vim-1, the full method achieved 70.5% identification for n=2, outperforming a competing method by at least 3%.