[Paper Review] Visio-Linguistic Brain Encoding

Subba Reddy Oota, Jashn Arora+3 more|arXiv (Cornell University)|Apr 18, 2022

Multimodal Machine Learning Applications9 citations

TL;DR

This paper investigates visio-linguistic brain encoding using vision-language Transformers (e.g., VisualBERT, LXMERT) to predict fMRI brain activity from visual and textual stimuli. It demonstrates that multi-modal Transformers significantly outperform single-modality models (CNNs, image Transformers) on BOLD5000 and Pereira datasets, establishing a new state-of-the-art and suggesting that linguistic processing may implicitly influence visual brain responses even during passive image viewing.

ABSTRACT

Enabling effective brain-computer interfaces requires understanding how the human brain encodes stimuli across modalities such as visual, language (or text), etc. Brain encoding aims at constructing fMRI brain activity given a stimulus. There exists a plethora of neural encoding models which study brain encoding for single mode stimuli: visual (pretrained CNNs) or text (pretrained language models). Few recent papers have also obtained separate visual and text representation models and performed late-fusion using simple heuristics. However, previous work has failed to explore: (a) the effectiveness of image Transformer models for encoding visual stimuli, and (b) co-attentive multi-modal modeling for visual and text reasoning. In this paper, we systematically explore the efficacy of image Transformers (ViT, DEiT, and BEiT) and multi-modal Transformers (VisualBERT, LXMERT, and CLIP) for brain encoding. Extensive experiments on two popular datasets, BOLD5000 and Pereira, provide the following insights. (1) To the best of our knowledge, we are the first to investigate the effectiveness of image and multi-modal Transformers for brain encoding. (2) We find that VisualBERT, a multi-modal Transformer, significantly outperforms previously proposed single-mode CNNs, image Transformers as well as other previously proposed multi-modal models, thereby establishing new state-of-the-art. The supremacy of visio-linguistic models raises the question of whether the responses elicited in the visual regions are affected implicitly by linguistic processing even when passively viewing images. Future fMRI tasks can verify this computational insight in an appropriate experimental setting.

Motivation & Objective

• To investigate the effectiveness of image and multi-modal Transformers in encoding fMRI brain activity from visual and textual stimuli.
• To determine whether joint visio-linguistic modeling improves brain encoding beyond single-modality approaches.
• To explore whether linguistic processing influences visual brain regions even during passive image viewing.
• To provide computational insights into how multi-modal representations align with human brain responses.

Proposed method

• The study employs vision-language Transformers (VisualBERT, LXMERT, CLIP) and image-specific Transformers (ViT, DEiT, BEiT) to encode stimuli.
• It uses cross-attention mechanisms to jointly model visual and linguistic features at multiple layers.
• fMRI activity is predicted using representations from the final layers of these models, with no manual layer selection.
• Experiments are conducted on two public fMRI datasets: BOLD5000 (purely visual stimuli) and Pereira (multi-modal stimuli).
• Performance is evaluated using Pearson correlation (PC) scores between predicted and actual fMRI responses across brain regions.
• The analysis includes ablation studies on concrete vs. abstract concepts and comparisons across brain networks (e.g., DMN, TP, visual areas).

Fig. 1: Brain encoding methodology. We use features from image/multi-modal Transformers (like ViT, VisualBERT and LXMERT) as input to the regression model to predict the fMRI activations for different brain regions. Brain encoding results are evaluated by computing 2V2 accuracy and Pearson correlati

Experimental results

Research questions

• RQ1 Can multi-modal Transformers like VisualBERT outperform single-modality models (CNNs, image Transformers) in fMRI brain encoding?
• RQ2 Does the inclusion of linguistic processing improve brain activity prediction even when only visual stimuli are presented?
• RQ3 How do different brain regions (e.g., visual, language, DMN) respond to visio-linguistic representations?
• RQ4 Is there a difference in encoding performance between concrete and abstract concepts?

Key findings

• VisualBERT achieves the highest performance on both BOLD5000 and Pereira datasets, setting a new state-of-the-art in visio-linguistic brain encoding.
• Multi-modal models like VisualBERT and LXMERT significantly outperform single-modality models (CNNs and image Transformers) on fMRI prediction tasks.
• Visual brain regions (e.g., Vision_Object, Vision_Face) show higher correlation with predicted fMRI responses than early visual areas.
• The concrete-train-abstract-test model yields better PC scores than the abstract-train-concrete-test model, indicating better learning from concrete concepts.
• Scene-selective regions (RSC, OPA) show higher correlation for COCO-Scenes, ImageNet-Scenes, and Scenes-Scenes tasks, especially when models are pre-trained on ImageNet or COCO.
• The study predicts that active tasks (e.g., naming or decision-making) would elicit stronger and more focused visual brain activation than passive viewing, suggesting a role for language in modulating visual processing.

Fig. 2: BOLD5000 Results: 2V2 (top figure) and Pearson correlation coefficient (bottom figure) between predicted and true responses across different brain regions using a variety of models. Results are averaged across all participants. VisualBERT perform the best.