[论文解读] Enabling hyperparameter optimization in sequential autoencoders for spiking neural data

Mohammad Reza Keshtkaran, Chethan Pandarinath|arXiv (Cornell University)|Jan 1, 2019

Neural dynamics and brain function被引用 8

一句话总结

本文引入了超参数优化（HPO）技术——样本验证和协同丢弃，以克服在稀疏、有限的脉冲神经数据上应用序列自编码器（SAEs）时的过拟合问题。通过实现有效的HPO，SAEs即使在小样本数据集上也能实现稳健性能，显著扩展了其从神经群体活动提取潜在动力学的能力。

ABSTRACT

Continuing advances in neural interfaces have enabled simultaneous monitoring of spiking activity from hundreds to thousands of neurons. To interpret these large-scale data, several methods have been proposed to infer latent dynamic structure from high-dimensional datasets. One recent line of work uses recurrent neural networks in a sequential autoencoder (SAE) framework to uncover dynamics. SAEs are an appealing option for modeling nonlinear dynamical systems, and enable a precise link between neural activity and behavior on a single-trial basis. However, the very large parameter count and complexity of SAEs relative to other models has caused concern that SAEs may only perform well on very large training sets. We hypothesized that with a method to systematically optimize hyperparameters (HPs), SAEs might perform well even in cases of limited training data. Such a breakthrough would greatly extend their applicability. However, we find that SAEs applied to spiking neural data are prone to a particular form of overfitting that cannot be detected using standard validation metrics, which prevents standard HP searches. We develop and test two potential solutions: an alternate validation method (“sample validation”) and a novel regularization method (“coordinated dropout”). These innovations prevent overfitting quite effectively, and allow us to test whether SAEs can achieve good performance on limited data through large-scale HP optimization. When applied to data from motor cortex recorded while monkeys made reaches in various directions, large-scale HP optimization allowed SAEs to better maintain performance for small dataset sizes. Our results should greatly extend the applicability of SAEs in extracting latent dynamics from sparse, multidimensional data, such as neural population spiking activity.

研究动机与目标

• 解决在有限脉冲神经数据上训练序列自编码器（SAEs）时面临的过拟合挑战。
• 开发可靠的超参数优化（HPO）方法，即使在标准验证指标无法检测SAEs过拟合的情况下也能有效工作。
• 通过新颖的验证和正则化技术，实现有效的HPO，将SAEs的应用范围扩展至小样本数据集。
• 测试SAEs是否可通过大规模HPO在稀疏、多维神经脉冲数据上保持高性能。

提出的方法

• 提出‘样本验证’作为标准验证的替代方法，即在保留样本而非时间分割上评估模型性能，以更好地检测SAEs中的过拟合。
• 引入‘协同丢弃’——一种正则化方法，通过在编码器和解码器中同步应用丢弃来改善SAEs的泛化能力。
• 利用所提出的验证和正则化技术，在多个SAE架构上进行大规模超参数优化（HPO）。
• 将优化后的SAEs应用于猴子在抓握任务期间运动皮层的神经脉冲数据，以评估其在小规模训练集上的性能。
• 利用SAEs从高维脉冲活动中推断低维潜在动力学，并在单次试验基础上将其与行为输出关联。

实验结果

研究问题

• RQ1 超参数优化能否提升序列自编码器（SAEs）在小样本脉冲神经活动数据集上的性能？
• RQ2 标准验证是否无法检测在脉冲数据上训练的SAEs中的过拟合？如果是，原因是什么？
• RQ3 样本验证和协同丢弃能否有效缓解SAEs在神经数据中的过拟合？
• RQ4 当超参数系统性优化时，SAEs在有限数据上能多大程度维持性能？

主要发现

• 样本验证成功检测到标准验证无法检测的SAEs过拟合，原因在于脉冲数据的时间结构。
• 协同丢弃通过在编码器和解码器之间同步正则化，显著改善了SAEs的泛化能力，减少了过拟合。
• 使用所提方法进行大规模超参数优化，使SAEs即使在小规模训练数据集上也能保持强劲性能。
• 在抓握任务中猴子运动皮层的脉冲数据上，优化后的SAEs在有限数据下捕捉潜在动力学的能力优于基线模型。
• 样本验证与协同丢弃的结合使SAEs能够可靠地应用于稀疏、多维神经数据，显著扩展了其实际应用价值。