[Paper Review] High-contrast "gaudy" images improve the training of deep neural network models of visual cortex

Benjamin R. Cowley, Jonathan W. Pillow|arXiv (Cornell University)|Jun 13, 2020

Image Processing Techniques and Applications59 references4 citations

TL;DR

This paper proposes 'gaudy' images—high-contrast, binarized versions of natural images—as a data-efficient training strategy for deep neural network (DNN) readout models of visual cortical neurons. By maximizing pixel variance through binary thresholding at the mean intensity, gaudy images overemphasize edges and high-contrast features, significantly reducing the number of training images needed to achieve high predictive accuracy, outperforming even active learning methods in some cases.

ABSTRACT

A key challenge in understanding the sensory transformations of the visual system is to obtain a highly predictive model of responses from visual cortical neurons. Deep neural networks (DNNs) provide a promising candidate for such a model. However, DNNs require orders of magnitude more training data than neuroscientists can collect from real neurons because experimental recording time is severely limited. This motivates us to find images that train highly-predictive DNNs with as little training data as possible. We propose gaudy images---high-contrast binarized versions of natural images---to efficiently train DNNs. In extensive simulation experiments, we find that training DNNs with gaudy images substantially reduces the number of training images needed to accurately predict the simulated responses of visual cortical neurons. We also find that gaudy images, chosen before training, outperform images chosen during training by active learning algorithms. Thus, gaudy images overemphasize features of natural images, especially edges, that are the most important for efficiently training DNNs. We believe gaudy images will aid in the modeling of visual cortical neurons, potentially opening new scientific questions about visual processing, as well as aid general practitioners that seek ways to improve the training of DNNs.

Motivation & Objective

• To reduce the amount of experimental neural recording data required to train accurate deep neural network (DNN) models of visual cortical responses.
• To identify image stimuli that maximize information gain during DNN training, especially under data scarcity.
• To test whether pre-selected, highly contrasted images can outperform adaptive, actively learned stimuli in training DNN readout models.
• To evaluate the utility of gaudy images for data augmentation and transfer learning in vision tasks.

Proposed method

• Generate gaudy images by binarizing each color channel of a natural image: set pixel intensity to 0 if below the image’s mean intensity, 255 if above.
• Train DNN readout models using gaudy images as input, with the goal of predicting responses from simulated visual cortical neurons.
• Compare performance of gaudy images against images selected via active learning (e.g., pool-based, core-set, ensemble-based methods) across multiple DNN architectures and pre-trained models (e.g., VGG19, ResNet-50).
• Use backpropagation to analyze gradients and identify which image features (e.g., edges) drive the most informative updates in the readout network.
• Assess data augmentation potential by comparing top-5 classification predictions of normal and gaudy images on ImageNet-pretrained DNNs.
• Control for class imbalance in image datasets by normalizing predicted class probabilities across gaudy and normal image sets.

Experimental results

Research questions

• RQ1 Can gaudy images, generated before training, outperform actively selected images in training DNN readout models of visual cortex responses?
• RQ2 To what extent do gaudy images reduce the number of training images needed to achieve high predictive accuracy in DNN models?
• RQ3 Which image features (e.g., edges, textures) are most informative for training DNN readout networks, and do gaudy images overemphasize them?
• RQ4 Are gaudy images suitable for data augmentation in object recognition tasks, given their structural similarity to natural images?
• RQ5 Do pre-trained DNNs maintain consistent classification outputs for gaudy images compared to their natural counterparts?

Key findings

• Gaudy images reduce the number of training images required to achieve high predictive accuracy in DNN readout models of visual cortical neurons, significantly outperforming standard natural images.
• Gaudy images outperform or match the performance of active learning algorithms in predicting responses from simulated V4 neurons, even though they are selected before training.
• The primary driver of improved training is the overemphasis of high-contrast edges in gaudy images, which generate large, informative gradients during backpropagation.
• A significant fraction (p < 0.001) of gaudy images retain at least one matching top-5 class prediction with their natural counterparts across multiple pre-trained DNNs, suggesting utility for data augmentation.
• Despite strong performance in classification similarity, gaudy images show a bias in predicted class probabilities, with higher concentration in specific classes (e.g., 400–1000), indicating potential class-specific misclassification risks.
• The results suggest that gaudy images are particularly effective at training DNNs when data is limited, and they may serve as a powerful, low-cost alternative to active learning in neuroscience and vision applications.