[论文解读] Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity

Felix Pei, Joel Ye+14 more|arXiv (Cornell University)|Sep 9, 2021

Neural dynamics and brain function参考文献 88被引用 44

一句话总结

引入 Neural Latents Benchmark (NLB) ’21，用以标准化在神经群体数据上对无监督潜变量模型（LVMs）的评估，覆盖多样的脑区、任务和数据集规模，以 co-smoothing 为主要度量并在 EvalAI 上托管基准。

ABSTRACT

Advances in neural recording present increasing opportunities to study neural activity in unprecedented detail. Latent variable models (LVMs) are promising tools for analyzing this rich activity across diverse neural systems and behaviors, as LVMs do not depend on known relationships between the activity and external experimental variables. However, progress with LVMs for neuronal population activity is currently impeded by a lack of standardization, resulting in methods being developed and compared in an ad hoc manner. To coordinate these modeling efforts, we introduce a benchmark suite for latent variable modeling of neural population activity. We curate four datasets of neural spiking activity from cognitive, sensory, and motor areas to promote models that apply to the wide variety of activity seen across these areas. We identify unsupervised evaluation as a common framework for evaluating models across datasets, and apply several baselines that demonstrate benchmark diversity. We release this benchmark through EvalAI. http://neurallatents.github.io

研究动机与目标

• Motivate standardized evaluation of latent variable models (LVMs) for neural population activity.
• Provide curated datasets spanning motor, sensory, and cognitive regions and varying dataset sizes.
• Define unsupervised evaluation framework with a robust primary metric (co-smoothing) and complementary metrics.
• Offer a reproducible pipeline with Neurodata Without Borders-formatted data and EvalAI-based evaluation.
• Baseline comparisons to establish performance benchmarks across model types.

提出的方法

• Curate four diverse neural spiking datasets (MC_Maze, MC_RTT, Area2_Bump, DMFC_RSG) with varying task demands.
• Adopt an unsupervised evaluation framework (co-smoothing) to predict held-out neural activity.
• Provide secondary metrics: PSTH match, forward prediction, and behavioral decoding where applicable.
• Compare five baseline LVM approaches (Smoothed spikes, GPFA, SLDS, AutoLFADS, Neural Data Transformer) to establish performance baselines.
• Utilize train/val/test splits with private test data on EvalAI to prevent overfitting and hyperparameter hacking.

实验结果

研究问题

• RQ1 How well can unsupervised latent variable models describe neural population activity across diverse brain regions and behaviors?
• RQ2 What is the relative performance of different LVM families (linear, nonlinear, deep) on held-out neuron/time predictions (co-smoothing) and secondary metrics?
• RQ3 How does model performance scale with dataset size and across datasets with varying numbers of neurons and firing rates?
• RQ4 Can the benchmark identify regimes where deep models (AutoLFADS, NDT) outperform traditional baselines, and where simpler models suffice?
• RQ5 How do evaluation metrics (co-smoothing vs. PSTH matching vs. forward prediction vs. behavioral decoding) align or diverge across datasets?

主要发现

• Co-smoothing is generally achievable across datasets, with deep models often outperforming baselines.
• Deep networks (AutoLFADS, NDT) show strong performance across multiple datasets, especially in more cognitive areas (DMFC_RSG) and larger datasets.
• Performance advantages of deep models are dataset-dependent, with some datasets showing smaller gaps.
• Benchmarked datasets scale from MC_Maze-L/M/S to MC_Maze and other tasks, illustrating dataset-size effects on LVM evaluation.
• GPFA and SLDS show variable performance across metrics, highlighting the importance of choosing appropriate evaluation criteria.
• Across datasets, deep models tend to offer the most consistent gains in co-smoothing performance.