[Paper Review] DreamCatcher: Revealing the Language of the Brain with fMRI using GPT Embedding

Subhrasankar Chatterjee, Debasis Samanta|arXiv (Cornell University)|Jun 16, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces DreamCatcher, a novel fMRI captioning framework that maps fMRI brain activity to 1536-dimensional GPT embeddings using a Representation Space Encoder (RSE) and generates natural language captions via a RevEmbedding Decoder. The method achieves promising performance in generating semantically coherent and contextually relevant captions from fMRI data, demonstrating the feasibility of using pre-trained language model embeddings as a brain representation space for decoding visual perception.

ABSTRACT

The human brain possesses remarkable abilities in visual processing, including image recognition and scene summarization. Efforts have been made to understand the cognitive capacities of the visual brain, but a comprehensive understanding of the underlying mechanisms still needs to be discovered. Advancements in brain decoding techniques have led to sophisticated approaches like fMRI-to-Image reconstruction, which has implications for cognitive neuroscience and medical imaging. However, challenges persist in fMRI-to-image reconstruction, such as incorporating global context and contextual information. In this article, we propose fMRI captioning, where captions are generated based on fMRI data to gain insight into the neural correlates of visual perception. This research presents DreamCatcher, a novel framework for fMRI captioning. DreamCatcher consists of the Representation Space Encoder (RSE) and the RevEmbedding Decoder, which transform fMRI vectors into a latent space and generate captions, respectively. We evaluated the framework through visualization, dataset training, and testing on subjects, demonstrating strong performance. fMRI-based captioning has diverse applications, including understanding neural mechanisms, Human-Computer Interaction, and enhancing learning and training processes.

Motivation & Objective

• To address the limitations of fMRI-to-image reconstruction, which often fails to capture global context and high-level scene semantics.
• To propose fMRI captioning as a complementary approach to image reconstruction, enabling interpretation of neural activity through natural language.
• To validate the use of 1536-dimensional GPT embeddings as a meaningful latent representation space for fMRI data.
• To develop a framework capable of generating semantically accurate and syntactically coherent captions directly from fMRI responses.

Proposed method

• The Representation Space Encoder (RSE) maps preprocessed fMRI vectors into a 1536-dimensional latent space using a neural network architecture trained to align with pre-trained GPT embeddings.
• The RevEmbedding Decoder is a one-to-many sequential model that generates natural language captions from the 1536-D GPT embedding representation.
• The framework uses a contrastive loss based on human-annotated captions to optimize the generation process, ensuring semantic alignment with reference descriptions.
• The model is trained and evaluated on the Natural Scenes Dataset (NSD), using fMRI data from 8 subjects exposed to visual stimuli.
• Principal Component Analysis (PCA) and t-SNE are applied to visualize the learned representation space and assess clustering of neural patterns.
• Evaluation metrics include METEOR, Sentence-level similarity, and Perplexity to assess caption quality and coherence.

Figure 1 : Illustrative example of current issues with fMRI-to-Image reconstruction. First reconstruction example successfully captures the low-level features however misses the high-level features. Second reconstruction example is adequate at a object-level replication, however misses the context i

Experimental results

Research questions

• RQ1 Can fMRI data be effectively mapped to a pre-trained language model's embedding space to enable meaningful caption generation?
• RQ2 Does the proposed DreamCatcher framework generate captions that are semantically and syntactically coherent with human-annotated reference captions?
• RQ3 Can the 1536-D GPT embedding space capture and differentiate salient features and contextual attributes of visual stimuli from fMRI signals?
• RQ4 How well does the framework generalize across subjects in generating contextually accurate descriptions from neural activity?
• RQ5 To what extent does the representation space learned by the RSE exhibit meaningful clustering of similar visual stimuli?

Key findings

• The DreamCatcher framework achieved favorable METEOR scores across all tested subjects, indicating strong alignment between generated and reference captions.
• The Sentence-level similarity metric demonstrated that generated captions were syntactically and semantically close to human-annotated descriptions, confirming coherence and relevance.
• Visualization using PCA and t-SNE revealed clear clustering of representations in the GPT embedding space, suggesting effective capture of underlying visual features and contextual relationships.
• The feasibility test confirmed that fMRI-based captioning is viable, with generated captions capturing salient objects and contextual information beyond low-level features.
• The RevEmbedding Decoder successfully generated one-to-many caption sequences from a single fMRI embedding, demonstrating the model’s capacity for rich, descriptive output.
• The framework outperformed fragment-based reconstruction methods in contextual fidelity, as evidenced by the absence of disconnected or incoherent features in generated captions.

Figure 2 : fMRI-Captioning: Subject is presented with an image stimulus and fMRI Neural Responses were captured. Given an fMRI response, the task is to predict captions based on the visual stimulus.