[Paper Review] Brain-like representational straightening of natural movies in robust feedforward neural networks

Tahereh Toosi, Elias B. Issa|arXiv (Cornell University)|Aug 26, 2023

Image Processing Techniques and Applications4 citations

TL;DR

This paper demonstrates that robust feedforward neural networks—trained via adversarial training or random smoothing—develop brain-like representational straightening in their feature space when processing natural movies, enabling linear interpolation to generate realistic intermediate frames. The key contribution is that this straightening emerges naturally from robustness to input noise, without explicit training on temporal dynamics or predictive objectives.

ABSTRACT

Representational straightening refers to a decrease in curvature of visual feature representations of a sequence of frames taken from natural movies. Prior work established straightening in neural representations of the primate primary visual cortex (V1) and perceptual straightening in human behavior as a hallmark of biological vision in contrast to artificial feedforward neural networks which did not demonstrate this phenomenon as they were not explicitly optimized to produce temporally predictable movie representations. Here, we show robustness to noise in the input image can produce representational straightening in feedforward neural networks. Both adversarial training (AT) and base classifiers for Random Smoothing (RS) induced remarkably straightened feature codes. Demonstrating their utility within the domain of natural movies, these codes could be inverted to generate intervening movie frames by linear interpolation in the feature space even though they were not trained on these trajectories. Demonstrating their biological utility, we found that AT and RS training improved predictions of neural data in primate V1 over baseline models providing a parsimonious, bio-plausible mechanism -- noise in the sensory input stages -- for generating representations in early visual cortex. Finally, we compared the geometric properties of frame representations in these networks to better understand how they produced representations that mimicked the straightening phenomenon from biology. Overall, this work elucidating emergent properties of robust neural networks demonstrates that it is not necessary to utilize predictive objectives or train directly on natural movie statistics to achieve models supporting straightened movie representations similar to human perception that also predict V1 neural responses.

Motivation & Objective

• To investigate whether robust neural networks can develop brain-like representational straightening in their feature space when processing natural movies.
• To determine if robustness to input noise—via adversarial training or random smoothing—can produce linear, invertible representations of natural movie sequences.
• To evaluate whether such robust networks better predict neural responses in primate primary visual cortex (V1) compared to standard feedforward networks.
• To explore the geometric properties of feature representations in robust networks and their relationship to biological straightening and dimensionality expansion.

Proposed method

• Trained ResNet-50 models using adversarial training (AT) and random smoothing (RS) with varying input noise levels (σ² = 0.1, 0.5, 1.0).
• Extracted feature representations from the final fully connected layer of trained networks for sequences of natural movie frames.
• Measured representational straightening by computing the curvature of trajectories formed by consecutive frames in the feature space.
• Evaluated invertibility by linearly interpolating between start and end frame features and decoding the interpolated features back into image space.
• Compared geometric properties such as expansion score (radial size of movie representations) and curvature to a standard ResNet50 baseline.
• Quantified model performance in predicting neural variance in primate V1 using correlation-based metrics, comparing robust and non-robust models.

Figure 1: Perceptual straightening of movie frames can be viewed as invertibility of latent representations for static images. Left: straightening of representations refers to a decrease in the curvature of the trajectory in representation space such as a neural population in the brain or human perc

Experimental results

Research questions

• RQ1 Can robust feedforward neural networks trained on static images develop representational straightening for natural movie sequences without explicit temporal supervision?
• RQ2 To what extent do adversarial training and random smoothing induce linear, invertible representations in the feature space of natural movies?
• RQ3 How do the geometric properties (curvature, expansion) of feature representations in robust networks compare to those in biological visual systems?
• RQ4 Does robustness to input noise improve the ability of neural network models to predict neural responses in primate V1 compared to standard classifiers?
• RQ5 Is there an optimal level of input noise (e.g., σ² = 0.5 in RS) that balances straightening and representation expansion to best match V1 data?

Key findings

• Robust networks trained via adversarial training and random smoothing exhibited significantly reduced curvature in feature trajectories of natural movies, matching the level of straightening observed in primate V1 and human perception.
• Linear interpolation between start and end frames in the feature space of robust networks generated plausible intermediate frames, demonstrating invertible, temporally predictive representations.
• The RS model with σ² = 0.5 achieved the best balance between high straightening and minimal representation contraction, outperforming other noise levels and baseline models in explaining V1 neural variance.
• Robust models, particularly RS with σ² = 0.5 and the best AT model, significantly outperformed standard ResNet50 in predicting neural responses in primate V1, indicating stronger biological plausibility.
• The expansion score of movie representations in the best RS model was highest relative to baseline ResNet50, suggesting that robustness supports dimensionality expansion consistent with cortical processing from retina to V1.
• The emergence of straightened representations in robust networks occurred without any training on temporal sequences or predictive objectives, indicating that robustness to noise is a sufficient mechanism for generating brain-like movie representations.

Figure 2: ANNs show straightening of representations when robustness to noise constraints (noise augmentation or adversarial attack) is added to their training. Measurements for straightening of movie sequences (from (Hénaff et al., 2019 ) , in each layer of ResNet50 architecture under different tra