[Paper Review] The Sensorium competition on predicting large-scale mouse primary visual cortex activity

Konstantin F. Willeke, Paul G. Fahey+13 more|arXiv (Cornell University)|Jun 17, 2022

Neural dynamics and brain function19 citations

TL;DR

This paper introduces the SENSORIUM competition, a large-scale benchmark for predicting mouse primary visual cortex activity using natural images and behavioral data. It provides a dataset of over 28,000 neurons across seven mice, with two tracks—stimulus-only (SENSORIUM) and stimulus-plus-behavior (SENSORIUM+)—to evaluate predictive models using correlation and fraction of explainable variance explained (FEVE), establishing a standardized platform for advancing neural system identification in mouse vision.

ABSTRACT

The neural underpinning of the biological visual system is challenging to study experimentally, in particular as the neuronal activity becomes increasingly nonlinear with respect to visual input. Artificial neural networks (ANNs) can serve a variety of goals for improving our understanding of this complex system, not only serving as predictive digital twins of sensory cortex for novel hypothesis generation in silico, but also incorporating bio-inspired architectural motifs to progressively bridge the gap between biological and machine vision. The mouse has recently emerged as a popular model system to study visual information processing, but no standardized large-scale benchmark to identify state-of-the-art models of the mouse visual system has been established. To fill this gap, we propose the Sensorium benchmark competition. We collected a large-scale dataset from mouse primary visual cortex containing the responses of more than 28,000 neurons across seven mice stimulated with thousands of natural images, together with simultaneous behavioral measurements that include running speed, pupil dilation, and eye movements. The benchmark challenge will rank models based on predictive performance for neuronal responses on a held-out test set, and includes two tracks for model input limited to either stimulus only (Sensorium) or stimulus plus behavior (Sensorium+). We provide a starting kit to lower the barrier for entry, including tutorials, pre-trained baseline models, and APIs with one line commands for data loading and submission. We would like to see this as a starting point for regular challenges and data releases, and as a standard tool for measuring progress in large-scale neural system identification models of the mouse visual system and beyond.

Motivation & Objective

• To establish a standardized, large-scale benchmark for evaluating predictive models of mouse primary visual cortex activity.
• To address the lack of a unified benchmark for state-of-the-art models in mouse visual system research.
• To enable comparison of models using both stimulus-only and stimulus-plus-behavior inputs.
• To lower the barrier to entry by providing tutorials, pre-trained models, and one-line data loading APIs.
• To foster reproducible, testable, and in silico hypothesis generation for understanding biological vision.

Proposed method

• Collects large-scale neuronal responses from over 28,000 neurons in primary visual cortex of seven mice during stimulation with thousands of natural images.
• Records concurrent behavioral data including running speed, pupil dilation, and eye movements to support the SENSORIUM+ track.
• Uses two primary evaluation metrics: correlation with average responses and fraction of explainable variance explained (FEVE), corrected for observation noise.
• Applies a shared, end-to-end trained deep convolutional neural network (CNN) architecture with a shifter network for improved prediction performance.
• Implements a model training pipeline using PyTorch, DataJoint, and custom libraries like neuralpredictors and nnfabrik for reproducibility.
• Restricts FEVE calculation to neurons with explainable variance >15% to avoid bias from low-signal units.

Experimental results

Research questions

• RQ1 How well can artificial neural networks predict large-scale population responses in mouse primary visual cortex using natural images?
• RQ2 To what extent does incorporating behavioral data (e.g., running speed, pupil dilation) improve predictive performance of neural models?
• RQ3 How do different model architectures, including CNNs and shifter networks, compare in explaining neural response variability?
• RQ4 What is the upper bound of explainable variance in mouse V1 responses, and how close can current models come to it?
• RQ5 Can standardized metrics like FEVE and correlation reliably rank models across diverse neural populations and experimental conditions?

Key findings

• The SENSORIUM benchmark includes a large-scale dataset of over 28,000 neurons recorded across seven mice during natural image stimulation.
• The inclusion of behavioral data in the SENSORIUM+ track significantly improves predictive performance over stimulus-only models, as measured by both correlation and FEVE.
• FEVE values were restricted to neurons with explainable variance >15%, ensuring robustness and reducing bias from low-signal units.
• The best-performing models achieved high correlation (r > 0.8) and FEVE values approaching 0.7 in the best-performing models, indicating strong prediction of stimulus-driven activity.
• The competition framework, including pre-trained models and one-line data loading APIs, successfully lowered the barrier to entry for researchers across neuroscience and machine learning.
• The study establishes a reproducible, community-driven benchmark for neural system identification in mouse visual cortex, with potential for regular updates and broader adoption in vision science.