[论文解读] A neural network trained to predict future video frames mimics critical properties of biological neuronal responses and perception

William Lotter, Gabriel Kreiman+1 more|arXiv (Cornell University)|May 28, 2018

Neural dynamics and brain function参考文献 45被引用 19

一句话总结

该论文表明，一种通过无标注数据训练以预测未来视频帧的循环预测神经网络，成功复现了生物视觉皮层反应的关键特性以及知觉运动错觉。该模型捕捉到了对静态刺激的动态神经反应，并通过自监督学习解释了复杂的视觉错觉，如闪光滞后效应和错觉轮廓，表明预测是大脑基本组织原则之一。

ABSTRACT

While deep neural networks take loose inspiration from neuroscience, it is an open question how seriously to take the analogies between artificial deep networks and biological neuronal systems. Interestingly, recent work has shown that deep convolutional neural networks (CNNs) trained on large-scale image recognition tasks can serve as strikingly good models for predicting the responses of neurons in visual cortex to visual stimuli, suggesting that analogies between artificial and biological neural networks may be more than superficial. However, while CNNs capture key properties of the average responses of cortical neurons, they fail to explain other properties of these neurons. For one, CNNs typically require large quantities of labeled input data for training. Our own brains, in contrast, rarely have access to this kind of supervision, so to the extent that representations are similar between CNNs and brains, this similarity must arise via different training paths. In addition, neurons in visual cortex produce complex time-varying responses even to static inputs, and they dynamically tune themselves to temporal regularities in the visual environment. We argue that these differences are clues to fundamental differences between the computations performed in the brain and in deep networks. To begin to close the gap, here we study the emergent properties of a previously-described recurrent generative network that is trained to predict future video frames in a self-supervised manner. Remarkably, the model is able to capture a wide variety of seemingly disparate phenomena observed in visual cortex, ranging from single unit response dynamics to complex perceptual motion illusions. These results suggest potentially deep connections between recurrent predictive neural network models and the brain, providing new leads that can enrich both fields.

研究动机与目标

• 探究自监督、循环预测神经网络是否能够模拟视觉皮层中观察到的动态神经反应。
• 检验此类模型是否能够解释根植于时间预测的知觉错觉，如闪光滞后效应和错觉轮廓。
• 确定在自然视频序列上进行无监督训练是否可生成与单神经元反应动力学及复杂知觉现象相似的表征。
• 探索深度学习中的预测编码原理是否可作为理解大脑功能的统一框架。

提出的方法

• 本研究采用PredNet，一种基于预测编码原理设计用于视频预测的深度循环神经网络架构。
• 该网络在大规模自然视频数据集（如KITTI）上端到端训练，使用下一帧预测目标，无需人工标注标签。
• 从各层提取内部表征，并使用标准神经科学指标与生物神经元反应进行比较。
• 通过测量刺激呈现和移除后的瞬态反应，以及对时间规律性的适应，分析神经反应动力学。
• 通过将模型预测与人类知觉报告对比，评估知觉错觉，特别是在闪光滞后和错觉轮廓任务中。
• 利用模型的预测误差解释瞬态神经反应，其中前向误差信号对应于观察到的神经瞬态。

实验结果

研究问题

• RQ1 在自然视频序列上训练的自监督循环神经网络，能否复现视觉皮层神经元对静态刺激的时变反应动力学？
• RQ2 预测性网络在多大程度上能够解释复杂的知觉错觉，如闪光滞后效应和错觉轮廓？
• RQ3 视频预测网络的内部表征是否与生物视觉处理的功能特性相似，即使未接受直接的生物学监督？
• RQ4 深度网络中的预测编码是否足以解释视觉中神经反应动力学与知觉现象？

主要发现

• PredNet模型对静态刺激产生了瞬态神经反应，包括起始和终止的爆发，与视觉皮层神经元的实验记录高度一致。
• 即使输入为静态，该模型在刺激呈现后仍表现出反应抑制，这是皮层反应动力学的典型特征。
• 在闪光滞后错觉中，模型预测的运动条带位置平均落后于实际条带1.4° ± 1.2°，显著低于帧复制方法的6°差异，表明存在预测性补偿。
• 预测闪光与运动条带之间的平均角度差为6.8° ± 2.0°，与人类观察者的感知滞后非常接近，表明模型在知觉上与统计预测对齐。
• 对于错觉轮廓，模型对错觉形状的反应振幅高于对物理线段的反应，且峰值反应出现时间延迟，与生物反应一致。
• 在所有层（E、A、R及层1和层2）中，单位对错觉轮廓相对于控制刺激的平均偏好为正值，表明对知觉性错觉特征具有稳健表征。