[论文解读] Brain-like representational straightening of natural movies in robust feedforward neural networks

Tahereh Toosi, Elias B. Issa|arXiv (Cornell University)|Aug 26, 2023

Image Processing Techniques and Applications被引用 4

一句话总结

该论文表明，通过对抗训练或随机平滑训练的鲁棒前馈神经网络在处理自然电影时，其特征空间中会自然产生类脑表征直线化，从而实现线性插值生成逼真的中间帧。关键贡献在于，这种直线化现象源于对输入噪声的鲁棒性，而无需显式训练时间动态或预测目标。

ABSTRACT

Representational straightening refers to a decrease in curvature of visual feature representations of a sequence of frames taken from natural movies. Prior work established straightening in neural representations of the primate primary visual cortex (V1) and perceptual straightening in human behavior as a hallmark of biological vision in contrast to artificial feedforward neural networks which did not demonstrate this phenomenon as they were not explicitly optimized to produce temporally predictable movie representations. Here, we show robustness to noise in the input image can produce representational straightening in feedforward neural networks. Both adversarial training (AT) and base classifiers for Random Smoothing (RS) induced remarkably straightened feature codes. Demonstrating their utility within the domain of natural movies, these codes could be inverted to generate intervening movie frames by linear interpolation in the feature space even though they were not trained on these trajectories. Demonstrating their biological utility, we found that AT and RS training improved predictions of neural data in primate V1 over baseline models providing a parsimonious, bio-plausible mechanism -- noise in the sensory input stages -- for generating representations in early visual cortex. Finally, we compared the geometric properties of frame representations in these networks to better understand how they produced representations that mimicked the straightening phenomenon from biology. Overall, this work elucidating emergent properties of robust neural networks demonstrates that it is not necessary to utilize predictive objectives or train directly on natural movie statistics to achieve models supporting straightened movie representations similar to human perception that also predict V1 neural responses.

研究动机与目标

• 研究鲁棒神经网络在处理自然电影时，是否能在其特征空间中发展出类脑表征直线化。
• 确定通过对抗训练或随机平滑实现的输入噪声鲁棒性，是否能在自然电影序列的特征空间中产生线性、可逆的表征。
• 评估此类鲁棒网络相较于标准前馈网络，是否能更优地预测灵长类初级视觉皮层（V1）的神经反应。
• 探索鲁棒网络中特征表征的几何特性及其与生物直线化和维度扩展的关系。

提出的方法

• 使用对抗训练（AT）和随机平滑（RS）在不同输入噪声水平（σ² = 0.1, 0.5, 1.0）下训练ResNet-50模型。
• 从训练后网络的最后全连接层提取自然电影帧序列的特征表征。
• 通过计算特征空间中连续帧形成的轨迹曲率来测量表征直线化程度。
• 通过线性插值起始帧和结束帧的特征并解码回图像空间，评估可逆性。
• 将扩展得分（电影表征的径向大小）和曲率等几何特性与标准ResNet50基线进行比较。
• 使用基于相关性的度量方法量化模型在预测灵长类V1神经变异性方面的性能，并比较鲁棒与非鲁棒模型。

Figure 1: Perceptual straightening of movie frames can be viewed as invertibility of latent representations for static images. Left: straightening of representations refers to a decrease in the curvature of the trajectory in representation space such as a neural population in the brain or human perc

实验结果

研究问题

• RQ1 在未显式进行时间监督的情况下，通过静态图像训练的鲁棒前馈神经网络能否在处理自然电影序列时发展出表征直线化？
• RQ2 对抗训练和随机平滑在多大程度上能诱导自然电影特征空间中线性、可逆的表征？
• RQ3 鲁棒网络中特征表征的几何特性（曲率、扩展）与生物视觉系统中的特性相比如何？
• RQ4 与标准分类器相比，对输入噪声的鲁棒性是否能提升神经网络模型预测灵长类V1神经反应的能力？
• RQ5 是否存在一个最优的输入噪声水平（例如RS中的σ² = 0.5），可在直线化与表征扩展之间实现最佳平衡，从而最好匹配V1数据？

主要发现

• 通过对抗训练和随机平滑训练的鲁棒网络在自然电影特征轨迹中表现出显著降低的曲率，其直线化程度与灵长类V1及人类感知中观察到的水平相当。
• 在鲁棒网络的特征空间中，对起始帧和结束帧进行线性插值可生成合理的中间帧，表明其具有线性、可逆的时间预测表征。
• 在σ² = 0.5时，RS模型在直线化与表征收缩最小化之间实现了最佳平衡，在解释V1神经变异性方面优于其他噪声水平和基线模型。
• 鲁棒模型，特别是σ² = 0.5的RS模型和最佳AT模型，在预测灵长类V1神经反应方面显著优于标准ResNet50，表明其具有更强的生物合理性。
• 最佳RS模型的电影表征扩展得分相对于基线ResNet50最高，表明鲁棒性支持与从视网膜到V1的皮层处理一致的维度扩展。
• 鲁棒网络中直线化表征的出现，完全未依赖时间序列或预测目标的训练，表明对噪声的鲁棒性足以生成类脑电影表征。

Figure 2: ANNs show straightening of representations when robustness to noise constraints (noise augmentation or adversarial attack) is added to their training. Measurements for straightening of movie sequences (from (Hénaff et al., 2019 ) , in each layer of ResNet50 architecture under different tra