[论文解读] The Dynamic Sensorium competition for predicting large-scale mouse visual cortex activity from videos

Polina Turishcheva, Paul G. Fahey+11 more|arXiv (Cornell University)|May 31, 2023

Neural dynamics and brain function被引用 11

一句话总结

本文介绍了 SENSORIUM 2023 基准测试——一个大规模、动态刺激–反应数据集和竞赛，旨在从视频中预测小鼠视觉皮层的单个神经元活动，包括带有行为输入的域内与域外(DOOD)评估。

ABSTRACT

Understanding how biological visual systems process information is challenging due to the complex nonlinear relationship between neuronal responses and high-dimensional visual input. Artificial neural networks have already improved our understanding of this system by allowing computational neuroscientists to create predictive models and bridge biological and machine vision. During the Sensorium 2022, we introduced benchmarks for vision models with static input. However, animals operate and excel in dynamic environments, making it crucial to study and understand how the brain functions under these conditions. Moreover, many biological theories, such as predictive coding, suggest that previous input is crucial for current input processing. Currently, there is no standardized benchmark to identify state-of-the-art dynamic models of the mouse visual system. To address this gap, we propose the Sensorium 2023 Benchmark Competition with dynamic input. It includes the collection of a new large-scale dataset from the primary visual cortex of ten mice, containing responses from over 78,000 neurons to over 2 hours of dynamic stimuli per neuron. Participants in the main benchmark track will compete to identify the best predictive models of neuronal responses for dynamic input. We will also host a bonus track in which submission performance will be evaluated on out-of-domain input, using withheld neuronal responses to dynamic input stimuli whose statistics differ from the training set. Both tracks will offer behavioral data along with video stimuli. As before, we will provide code, tutorials, and strong pre-trained baseline models to encourage participation. We hope this competition will continue to strengthen the accompanying Sensorium benchmarks collection as a standard tool to measure progress in large-scale neural system identification models of the entire mouse visual hierarchy and beyond.

研究动机与目标

• 为小鼠视觉系统的动态神经预测建立标准化基准并提供形式化框架。
• 提供一个包含多只小鼠的动态视觉刺激、神经反应和行为的大规模数据集。
• 鼓励开发能处理时间动态并能推广到域外刺激的预测模型。
• 将行为变量作为模型输入，以反映对神经反应的调节因素。

提出的方法

• 提出一个动态预测建模框架 f_theta(x,b)，将视频刺激 x 与行为 b 映射到随时间变化的神经反应 r。
• 提供一个大规模数据集：five mice, 38,819 neurons, ~600 minutes of dynamic stimuli，以及训练/验证/实时测试/最终测试分区。
• 描述基线模型架构：GRU 基线，核心为 2D CNN，Gaussian 读出；3D 因式分解基线，核心为动态核，Gaussian 读出；集成基线。
• 提供一个起始工具包，含教程、代码和 API，用于加载数据和训练模型。
• 定义评估指标：主轨道的自然视频最终测试集上的单次试验相关性；奖金轨道在五个 OOD 测试集上的同一指标；还报告与平均值的相关性。

实验结果

研究问题

• RQ1 使用视频和行为作为输入时，在小鼠单个细胞神经反应对动态自然电影刺激的预测性能上限是多少？
• RQ2 动态模型在预测随时间演变的神经活动方面，与静态或非时序基线相比表现如何？
• RQ3 将行为变量（ locomotion, pupil size, eye position ）纳入是否能提高预测准确性？
• RQ4 在域内自然视频上训练的模型是否能够推广到具有不同统计特征的域外刺激？
• RQ5 哪些架构和训练策略最能捕捉小鼠视觉皮层的时空动态？

主要发现

• 发布了一个包含 five mice 共 38,819 个神经元、覆盖 ~600 分钟动态刺激的大规模数据集。
• 基线模型（GRU、3D Factorized、以及集成）在性能上表现不一，通常集成基线在留出测试数据上优于单一基线。
• 竞赛强调动态输入（视频）而非静态图像，并为域外评估设有奖金轨道。
• 将行为变量作为输入纳入预测模型，反映对神经活动的调节作用。
• 性能评估在主轨道的自然视频最终测试集上采用单次试验相关性，并在多个 OOD 刺激上进行评估（奖金轨道）。
• 论文提供起始工具包、教程以及预训练基线，方便参与和实现跨团队标准化。