[Paper Review] Learning shared neural manifolds from multi-subject FMRI data

Jessie Huang, Erica L. Busch+7 more|arXiv (Cornell University)|Dec 22, 2021

EEG and Brain-Computer Interfaces4 citations

TL;DR

This paper proposes MRMD-AE, a deep neural network that learns a shared, low-dimensional manifold from multi-subject fMRI data by encoding individual brain activity into a common latent space while preserving subject-specific features through multiple decoders. The method improves stimulus classification, enables zero-shot mapping of unseen data points, and enhances cross-subject fMRI translation, outperforming linear baselines and prior methods like SRM and PHATE in accuracy and alignment.

ABSTRACT

Functional magnetic resonance imaging (fMRI) is a notoriously noisy measurement of brain activity because of the large variations between individuals, signals marred by environmental differences during collection, and spatiotemporal averaging required by the measurement resolution. In addition, the data is extremely high dimensional, with the space of the activity typically having much lower intrinsic dimension. In order to understand the connection between stimuli of interest and brain activity, and analyze differences and commonalities between subjects, it becomes important to learn a meaningful embedding of the data that denoises, and reveals its intrinsic structure. Specifically, we assume that while noise varies significantly between individuals, true responses to stimuli will share common, low-dimensional features between subjects which are jointly discoverable. Similar approaches have been exploited previously but they have mainly used linear methods such as PCA and shared response modeling (SRM). In contrast, we propose a neural network called MRMD-AE (manifold-regularized multiple decoder, autoencoder), that learns a common embedding from multiple subjects in an experiment while retaining the ability to decode to individual raw fMRI signals. We show that our learned common space represents an extensible manifold (where new points not seen during training can be mapped), improves the classification accuracy of stimulus features of unseen timepoints, as well as improves cross-subject translation of fMRI signals. We believe this framework can be used for many downstream applications such as guided brain-computer interface (BCI) training in the future.

Motivation & Objective

• To address the challenge of high noise and inter-subject variability in fMRI data by learning a shared, low-dimensional latent space that captures common neural responses across individuals.
• To enable robust classification of stimuli from fMRI data by disentangling shared neural patterns from individual-specific noise.
• To support zero-shot generalization by learning an extensible manifold that maps previously unseen timepoints into the shared latent space.
• To facilitate cross-subject fMRI signal translation without requiring paired data, enabling prediction of one subject's brain activity from another’s.
• To improve upon linear methods like PCA and SRM by leveraging nonlinear, manifold-regularized deep autoencoders with shared and subject-specific components.

Proposed method

• The model uses a shared encoder to map multi-subject fMRI data into a common low-dimensional latent space, preserving shared stimulus-related features while filtering out individual noise.
• It employs multiple subject-specific decoders to reconstruct individual fMRI signals from the shared latent representation, enabling disentanglement of shared and individual components.
• The framework incorporates manifold regularization via PHATE-based loss to preserve the intrinsic geometry of neural activity across subjects.
• A latent space alignment penalty (λ > 0) is introduced to enforce structural similarity across subjects in the shared space, reducing batch effects.
• A translation loss is applied during training to explicitly optimize for cross-subject signal translation, improving performance in zero-shot settings.
• The model is trained end-to-end using a combined loss function balancing reconstruction, manifold regularization, and alignment/translation penalties.

Experimental results

Research questions

• RQ1 Can a deep autoencoder with shared and subject-specific components learn a common neural manifold that improves stimulus classification beyond linear methods?
• RQ2 Can the shared latent space support zero-shot mapping of unseen fMRI timepoints, enabling generalization to new data?
• RQ3 To what extent does manifold regularization and latent space alignment improve cross-subject fMRI signal translation?
• RQ4 How does the performance of MRMD-AE compare to linear baselines (e.g., PCA, SRM) and nonlinear methods (e.g., PHATE) in classification and translation tasks?
• RQ5 Can the model be trained to perform cross-subject translation without explicit paired data, leveraging only shared stimulus responses?

Key findings

• MRMD-AE achieved a classification accuracy of 0.5837 ± 0.0209 on unseen timepoints, outperforming the next best method (AE at 0.5613 ± 0.0220).
• The model demonstrated superior cross-subject translation, achieving a mean squared error (MSE) of 0.1933 and correlation of 1.3626 on early-visual ROI when using latent space alignment (λ > 0), outperforming untrained baselines.
• With explicit translation loss, MRMD-AE achieved the lowest translation MSE of 0.1647 on the pmc ROI, surpassing even SRM and PCA in performance.
• PHATE visualization confirmed that the shared latent space significantly reduced batch effects compared to raw fMRI data, with decreasing Earth Mover’s Distance (EMD) as alignment penalty λ increased.
• The shared latent space enabled extensible manifold learning, allowing mapping of new, unseen data points into the common representation without retraining.
• Incorporating latent space alignment penalties (λ > 0) improved both classification and translation performance, demonstrating the importance of structural invariance in the shared space.