[Paper Review] MindGPT: Interpreting What You See with Non-invasive Brain Recordings

Jiaxuan Chen, Qi Yu+2 more|arXiv (Cornell University)|Sep 27, 2023

Neural dynamics and brain functionNeuroscience3 citations

TL;DR

MindGPT is a non-invasive neural decoder that translates fMRI brain signals into natural language descriptions of visual stimuli by leveraging a vision-language contrastive encoder with cross-attention and GPT-2 for end-to-end semantic alignment. It achieves high-fidelity semantic reconstruction of visual content using only higher visual cortex (HVC) signals, demonstrating that HVC carries most semantic information and that visual cues guide decoding through attention-driven saliency maps.

ABSTRACT

Decoding of seen visual contents with non-invasive brain recordings has important scientific and practical values. Efforts have been made to recover the seen images from brain signals. However, most existing approaches cannot faithfully reflect the visual contents due to insufficient image quality or semantic mismatches. Compared with reconstructing pixel-level visual images, speaking is a more efficient and effective way to explain visual information. Here we introduce a non-invasive neural decoder, termed as MindGPT, which interprets perceived visual stimuli into natural languages from fMRI signals. Specifically, our model builds upon a visually guided neural encoder with a cross-attention mechanism, which permits us to guide latent neural representations towards a desired language semantic direction in an end-to-end manner by the collaborative use of the large language model GPT. By doing so, we found that the neural representations of the MindGPT are explainable, which can be used to evaluate the contributions of visual properties to language semantics. Our experiments show that the generated word sequences truthfully represented the visual information (with essential details) conveyed in the seen stimuli. The results also suggested that with respect to language decoding tasks, the higher visual cortex (HVC) is more semantically informative than the lower visual cortex (LVC), and using only the HVC can recover most of the semantic information. The code of the MindGPT model will be publicly available at https://github.com/JxuanC/MindGPT.

Motivation & Objective

• To develop a non-invasive brain-computer interface that translates visual perception into natural language using fMRI.
• To address the limitations of image reconstruction methods, which often produce blurry or semantically mismatched outputs.
• To investigate whether brain activity can be used to generate semantically accurate, well-formed language descriptions of visual stimuli.
• To explore the role of visual attention and saliency in guiding semantic decoding from fMRI signals.
• To determine which brain regions—especially in the visual cortex—contribute most to semantic representation in language decoding.

Proposed method

• A CLIP-guided fMRI encoder with cross-attention layers is used to align brain activity patterns with visual and linguistic embeddings in a shared latent space.
• The model employs GPT-2 as a text generator, conditioned on fMRI-encoded representations to produce fluent, coherent language sequences.
• Cross-attention mechanisms enable the model to attend to relevant brain regions that correspond to specific semantic concepts in the generated text.
• The framework is trained end-to-end using contrastive learning to align fMRI representations with corresponding CLIP-embedded image patches and GPT-2-generated captions.
• Visual saliency is inferred via cosine similarity between fMRI-encoded class embeddings and CLIP patch embeddings, without any supervision.
• The model is evaluated using human-annotated image captions and compared against visual reconstruction baselines like VQ-fMRI and MinD-Vis.

Experimental results

Research questions

• RQ1 Can fMRI signals alone, without invasive recording, be used to generate accurate and semantically meaningful natural language descriptions of visual stimuli?
• RQ2 Which regions of the visual cortex—lower (LVC) or higher (HVC)—contribute more to semantic decoding in language generation?
• RQ3 To what extent do visual attention mechanisms, inferred from fMRI-CLIP similarity maps, guide the semantic content of generated language?
• RQ4 Can the model generalize across diverse visual stimuli using only limited fMRI training data and no explicit attention supervision?
• RQ5 How do brain-derived representations relate to amodal semantic maps, and can they be used to reconstruct conceptual meaning in language?

Key findings

• MindGPT successfully generates natural language descriptions that accurately reflect essential visual details of stimuli, outperforming image reconstruction baselines in semantic fidelity.
• The higher visual cortex (HVC) alone contains sufficient neural information to recover most of the semantic content of visual stimuli, outperforming the lower visual cortex (LVC).
• Visual saliency maps inferred from fMRI-CLIP similarity show strong alignment with decoded language content, such as 'piano', 'airplane', or 'beach', indicating attention-driven decoding.
• Decoding errors, such as incorrect color attribution ('black and white photo'), can be traced back to mislocalized attention in fMRI representations, confirming the role of saliency in semantic bias.
• The latent representations of MindGPT exhibit locality-sensitive properties, aligning with known neuroscientific findings on hierarchical visual processing.
• The model demonstrates emergent ability to capture task-relevant visual cues even with limited training data, suggesting robust generalization of semantic representations.