[Paper Review] A General Framework for Revealing Human Mind with auto-encoding GANs

Pan Wang, Rui Zhou+7 more|arXiv (Cornell University)|Feb 10, 2021

Neural dynamics and brain function14 references4 citations

TL;DR

This paper proposes an end-to-end brain decoding framework that translates brain activity into visual images using auto-encoding GANs, aligning latent representations of EEG and fMRI signals with image features. The method achieves high-fidelity mental image reconstruction, with fMRI-based reconstruction outperforming EEG due to better signal resolution, demonstrating near-100% correct rate in image similarity comparisons using SSIM and MSE.

ABSTRACT

Addressing the question of visualising human mind could help us to find regions that are associated with observed cognition and responsible for expressing the elusive mental image, leading to a better understanding of cognitive function. The traditional approach treats brain decoding as a classification problem, reading the mind through statistical analysis of brain activity. However, human thought is rich and varied, that it is often influenced by more of a combination of object features than a specific type of category. For this reason, we propose an end-to-end brain decoding framework which translates brain activity into an image by latent space alignment. To find the correspondence from brain signal features to image features, we embedded them into two latent spaces with modality-specific encoders and then aligned the two spaces by minimising the distance between paired latent representations. The proposed framework was trained by simultaneous electroencephalogram and functional MRI data, which were recorded when the subjects were viewing or imagining a set of image stimuli. In this paper, we focused on implementing the fMRI experiment. Our experimental results demonstrated the feasibility of translating brain activity to an image. The reconstructed image matches image stimuli approximate in both shape and colour. Our framework provides a promising direction for building a direct visualisation to reveal human mind.

Motivation & Objective

• To address the limitations of traditional brain decoding as a classification task by enabling end-to-end image reconstruction from brain activity.
• To establish a direct mapping between brain signal features and image features through latent space alignment.
• To evaluate the feasibility of reconstructing mental images from both EEG and fMRI signals using a unified generative framework.
• To compare the performance of EEG and fMRI in reconstructing visual mental content, focusing on image quality and feature fidelity.

Proposed method

• The framework uses modality-specific encoders to embed brain signals (EEG/fMRI) and images into shared latent spaces.
• Latent representations of brain activity and images are aligned by minimizing the L2 distance between paired representations.
• A variational auto-encoder is pre-trained for image feature learning and image generation, forming the generative backbone.
• A brain signal encoder is trained to map fMRI and EEG signals to the same latent space as image features.
• The model is trained on simultaneous EEG-fMRI data collected during image viewing and mental imagery tasks.
• Image reconstruction is performed by decoding the aligned latent representation of brain activity into a visual image.

Experimental results

Research questions

• RQ1 Can brain activity be directly translated into a visual image using end-to-end deep generative modeling?
• RQ2 How well can latent space alignment between brain signals and image features reconstruct mental images?
• RQ3 What is the relative performance of EEG versus fMRI in reconstructing visual mental content?
• RQ4 To what extent does the reconstructed image preserve shape and color information compared to the original stimulus?

Key findings

• The fMRI-based framework achieved a pairwise image similarity correct rate of approximately 80% in SSIM comparison, indicating high-quality reconstruction of shape and color.
• The EEG-based framework achieved a correct rate of around 60% in similarity comparison, showing detectable but blurry image reconstruction.
• The alpha-GAN baseline, using real images as input, achieved a near 100% correct rate in image similarity comparison, confirming the model's capability.
• Despite lower image quality, EEG signals showed a higher classification rate than fMRI, suggesting EEG may be more suitable for category-level decoding.
• The framework successfully reconstructed mental images from both fMRI and EEG signals, demonstrating the feasibility of direct visualisation of human thought.