[论文解读] Towards robust vision by multi-task learning on monkey visual cortex

Shahd Safarani, Arne Nix+6 more|arXiv (Cornell University)|Jul 29, 2021

Visual Attention and Saliency Detection参考文献 37被引用 10

一句话总结

本文提出一种多任务学习框架，联合训练深度神经网络以实现图像分类和猕猴初级视觉皮层（V1）神经反应预测，利用生物诱导偏差提升对图像失真的鲁棒性。尽管仅在干净图像上进行训练，该V1正则化网络在分布外泛化方面表现显著提升，接近在受损数据上训练的Oracle模型性能，并展现出更具生物特性的、对显著性敏感的特征表征。

ABSTRACT

Deep neural networks set the state-of-the-art across many tasks in computer vision, but their generalization ability to image distortions is surprisingly fragile. In contrast, the mammalian visual system is robust to a wide range of perturbations. Recent work suggests that this generalization ability can be explained by useful inductive biases encoded in the representations of visual stimuli throughout the visual cortex. Here, we successfully leveraged these inductive biases with a multi-task learning approach: we jointly trained a deep network to perform image classification and to predict neural activity in macaque primary visual cortex (V1). We measured the out-of-distribution generalization abilities of our network by testing its robustness to image distortions. We found that co-training on monkey V1 data leads to increased robustness despite the absence of those distortions during training. Additionally, we showed that our network's robustness is very close to that of an Oracle network where parts of the architecture are directly trained on noisy images. Our results also demonstrated that the network's representations become more brain-like as their robustness improves. Using a novel constrained reconstruction analysis, we investigated what makes our brain-regularized network more robust. We found that our co-trained network is more sensitive to content than noise when compared to a Baseline network that we trained for image classification alone. Using DeepGaze-predicted saliency maps for ImageNet images, we found that our monkey co-trained network tends to be more sensitive to salient regions in a scene, reminiscent of existing theories on the role of V1 in the detection of object borders and bottom-up saliency. Overall, our work expands the promising research avenue of transferring inductive biases from the brain, and provides a novel analysis of the effects of our transfer.

研究动机与目标

• 探究联合训练深度网络进行图像分类和猕猴V1神经反应预测，是否能提升其对图像失真的鲁棒性。
• 检验神经共训练假说，即类脑表征可为人工网络提供更好的归纳偏差以实现泛化。
• 分析类脑正则化表征与标准分类网络在敏感性和不变性方面的差异。
• 评估提升的鲁棒性是否与对显著图像特征的敏感性相关，如V1功能理论所预测。

提出的方法

• 使用相同的自然图像刺激，联合训练深度神经网络以执行图像分类和预测猕猴V1的尖峰反应。
• 在图像分类头和神经反应预测头之间使用共享表征分支，以强制实施多任务学习。
• 仅在干净的ImageNet图像上训练模型，不使用任何涉及失真的数据增强。
• 将多任务模型的鲁棒性与仅分类的基线模型及在受损图像上训练的Oracle模型进行比较。
• 应用一种新型约束重建分析，以探测学习表征中的特征敏感性和不变性。
• 使用DeepGaze预测的显著性图，评估网络是否聚焦于显著的、与行为相关的图像区域。

实验结果

研究问题

• RQ1 尽管在训练期间未接触任何失真数据，使用猴子V1神经数据进行多任务学习是否能提升深度网络对图像失真的鲁棒性？
• RQ2 V1正则化网络的鲁棒性与仅在受损图像上训练的Oracle模型相比如何？
• RQ3 类脑正则化网络中，哪些特征敏感性和不变性变化导致了鲁棒性的提升？
• RQ4 V1正则化网络的表征在多大程度上与显著性图对齐，表明其在自下而上注意中的作用？

主要发现

• 在tiny ImageNet上，与仅分类的基线相比，联合训练图像分类和猕猴V1反应的多任务网络在15种常见图像失真上表现出显著更高的鲁棒性。
• V1正则化网络的鲁棒性与仅部分网络在受损数据上训练的Oracle模型相当，表明强大的归纳偏差迁移。
• 约束重建分析显示，V1正则化网络对图像内容更敏感，对噪声更不敏感，表明其具有更好的不变性。
• 随着鲁棒性的提升，网络的特征表征变得更像大脑表征，与V1中已知的神经反应模式对齐更强。
• V1正则化网络对DeepGaze显著性图预测的显著图像区域表现出更高的敏感性，与V1在检测物体边界和自下而上显著性中的作用一致。
• 结果支持神经共训练假说，并表明生物神经数据可作为人工网络的强大正则化器，无需数据增强即可提升泛化能力。