[论文解读] Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex

Drew Linsley, Iván Rodríguez+5 more|arXiv (Cornell University)|Jun 6, 2023

Neural dynamics and brain function被引用 11

一句话总结

本文表明，随着 DNN 在 ImageNet 上性能提升，它们在预测 IT 神经反应方面变得更差；使用 neural harmonizer 进行训练能够让表示与人类对齐并恢复神经预测性。

ABSTRACT

One of the most impactful findings in computational neuroscience over the past decade is that the object recognition accuracy of deep neural networks (DNNs) correlates with their ability to predict neural responses to natural images in the inferotemporal (IT) cortex. This discovery supported the long-held theory that object recognition is a core objective of the visual cortex, and suggested that more accurate DNNs would serve as better models of IT neuron responses to images. Since then, deep learning has undergone a revolution of scale: billion parameter-scale DNNs trained on billions of images are rivaling or outperforming humans at visual tasks including object recognition. Have today's DNNs become more accurate at predicting IT neuron responses to images as they have grown more accurate at object recognition? Surprisingly, across three independent experiments, we find this is not the case. DNNs have become progressively worse models of IT as their accuracy has increased on ImageNet. To understand why DNNs experience this trade-off and evaluate if they are still an appropriate paradigm for modeling the visual system, we turn to recordings of IT that capture spatially resolved maps of neuronal activity elicited by natural images. These neuronal activity maps reveal that DNNs trained on ImageNet learn to rely on different visual features than those encoded by IT and that this problem worsens as their accuracy increases. We successfully resolved this issue with the neural harmonizer, a plug-and-play training routine for DNNs that aligns their learned representations with humans. Our results suggest that harmonized DNNs break the trade-off between ImageNet accuracy and neural prediction accuracy that assails current DNNs and offer a path to more accurate models of biological vision.

研究动机与目标

• 评估现代高准确度的 DNN 是否更好地模拟对自然图像的 inferotemporal (IT) 皮层反应。
• 探究为何当 DNN 规模扩大时，任务优化会导致与 IT 的对齐丧失。
• 评估是否采用替代的训练方案或生物学约束能够提高 IT 的预测性。
• 提出并测试一种训练方案（neural harmonizer）以将 DNN 表征与人类视觉特征和 IT 反应对齐。

提出的方法

• 评估 135 种多样化的 DNN（CNNs、ViTs、自监督、鲁棒性训练）在 ImageNet 或其他数据上进行预训练后，使用 Brain-Score 风格的神经预测进行评估。
• 记录两只猴子对高分辨率自然图像的 IT 大脑神经元的空间分辨响应。
• 在 DNN 上训练并测试 neural harmonizer，以将人类特征重要性图与 DNN 表征对齐。
• 使用部分最小二乘回归将 DNN 单元活动映射到 IT 神经元反应并计算神经预测性。
• 应用基于 CRAFT 的特征分解来解释在和标准 DNN 相比，和谐化模型驱动 IT 反应的图像特征。
• 比较模型与时间时隙上的神经预测性，并评估 IT 与 DNN 之间的特征对齐。

Figure 1 : Deep neural networks (DNNs) are becoming progressively worse models of inferior temporal (IT) cortex as they grow more accurate on ImageNet. Experimental data shown here is taken from Brain-score.org , and each experiment utilized different stimuli. The 104 dots in each panel depict the I

实验结果

研究问题

• RQ1 更高的 ImageNet 精度是否在现代 DNN 中与更好的 IT 神经预测性相关？
• RQ2 ImageNet 训练的 DNN 依赖于哪些特征，这些特征与自然图像的 IT 编码有何不同？
• RQ3 将 DNN 表征与人类感知特征（neural harmonizer）对齐是否可以在不牺牲精度的情况下改善 IT 预测性？
• RQ4 生物学对齐的训练方案是否能缓解对象识别与神经数据之间的不匹配？

主要发现

• 在 ImageNet 上预训练的 DNN 在 ImageNet 精度增加时，预测 IT 神经元反应的准确性变得更低。
• 如自监督学习或对抗鲁棒性等训练方法并不能解决 IT 预测性的权衡。
• 和谐化的 DNN（hDNN）在两只猴子的 PL 和 ML 区域显著提高了 IT 预测性。
• 和谐模型揭示了驱动 IT 活动的特征，与人类判断（如面部部位）对齐，而不是背景特征。
• 基于 CRAFT 的分析表明，和谐化模型提供了可验证、可解释的关于 IT 特征选择性的问题假说。
• 和谐化模型打破了在标准 DNN 中观察到的 ImageNet 精度与神经预测精度之间的帕累托前沿。

Figure 2 : IT recordings that reveal spatial maps of neuronal responses to complex natural images offer unprecedented insights into their feature selectivity [ 6 ] . (a) Neurons in posterior (PL) and/or medial (ML) lateral IT in two animals were localized using functional magnetic resonance imaging