[Paper Review] V1T: large-scale mouse V1 response prediction using a Vision Transformer

Bryan M. Li, Isabel Maria Cornacchia+2 more|arXiv (Cornell University)|Feb 6, 2023

Neural dynamics and brain functionNeuroscience3 citations

TL;DR

This paper introduces V1T, a Vision Transformer-based model that predicts mouse primary visual cortex (V1) neural responses to natural visual stimuli with state-of-the-art performance. By leveraging self-attention mechanisms to learn shared visual and behavioral representations across animals, V1T achieves a 12.7% improvement over prior convolutional models and reveals biologically relevant features through attention map correlations with receptive fields and behavioral variables like pupil center and running speed.

ABSTRACT

Accurate predictive models of the visual cortex neural response to natural visual stimuli remain a challenge in computational neuroscience. In this work, we introduce V1T, a novel Vision Transformer based architecture that learns a shared visual and behavioral representation across animals. We evaluate our model on two large datasets recorded from mouse primary visual cortex and outperform previous convolution-based models by more than 12.7% in prediction performance. Moreover, we show that the self-attention weights learned by the Transformer correlate with the population receptive fields. Our model thus sets a new benchmark for neural response prediction and can be used jointly with behavioral and neural recordings to reveal meaningful characteristic features of the visual cortex.

Motivation & Objective

• To develop a scalable, data-driven predictive model for mouse V1 neural responses using large-scale neural recordings.
• To evaluate whether Vision Transformers (ViTs) can outperform convolutional neural networks (CNNs) in modeling mouse V1 responses.
• To investigate whether self-attention mechanisms in ViTs reflect biological properties such as receptive fields and behavioral modulation.
• To integrate behavioral variables—such as running speed, pupil dilation, and pupil center—into the predictive framework to improve response modeling.
• To provide a new in silico framework for probing cortical computations in the visual system using attention visualization and ablation studies.

Proposed method

• A Vision Transformer (ViT) is used as the shared core representation module, replacing traditional CNNs in the core-readout architecture.
• The model processes video stimuli through patch embedding and learns global dependencies via multi-head self-attention mechanisms.
• Behavioral variables (running speed, pupil dilation, pupil center, and its derivative) are integrated into the latent embedding via a learnable B-MLP module.
• A linear readout head maps the shared ViT features to individual neuron responses for each animal, enabling end-to-end training.
• The model is trained on two large-scale mouse V1 datasets with thousands of neurons, using correlation-based loss to maximize prediction accuracy.
• Attention visualization and ablation studies are used to interpret the model’s behavior and assess the influence of specific behavioral variables.

Figure 1 : Illustration of the V 1 T block architecture.

Experimental results

Research questions

• RQ1 Can a Vision Transformer-based architecture outperform convolutional neural networks in predicting mouse V1 neural responses to natural visual stimuli?
• RQ2 Do the self-attention weights in the ViT model correlate with known biological features such as population receptive fields or retinotopy?
• RQ3 To what extent can behavioral variables like running speed and pupil dynamics improve the predictive performance of neural response models?
• RQ4 How does the performance of the ViT model scale with increasing amounts of training data compared to CNN-based models?
• RQ5 Can the attention mechanism in a ViT model reveal insights into the functional organization of mouse V1, such as attentional modulation or spatial tuning?

Key findings

• V1T achieves a single-trial correlation of 0.428 and 0.444 on two held-out test sets, outperforming prior convolution-based models by over 12.7% in prediction performance.
• The model’s self-attention maps show significant correlation with the pupil center of the animal, indicating that attention heads reflect retinotopic organization and fixation-related processing.
• V1T demonstrates superior data efficiency, with performance that continues to improve with more data, outperforming CNNs in high-data regimes.
• The model achieves second-place performance on the leaderboard for mouse V1 prediction, surpassing all models not using temporal response trends.
• Incorporating behavioral variables such as running speed and pupil dilation enhances predictive accuracy, confirming their modulatory role in V1 responses.
• The B-MLP module enables effective integration of scalar behavioral signals into the ViT’s latent space, supporting multimodal input fusion without architectural overhauls.

Figure 2 : Prediction performance when trained with 500, 1500, 2500, 3500 and (all) 4500 samples per animal in Dataset S . The models were each trained with 30 different random seeds. The error bar shows the standard deviations of the repeated experiments, and the statistical difference (two-sided t