[Paper Review] The Origins and Prevalence of Texture Bias in Convolutional Neural Networks

Katherine L. Hermann, Ting Chen+1 more|arXiv (Cornell University)|Nov 20, 2019

Adversarial Robustness in Machine Learning87 references120 citations

TL;DR

This paper investigates why ImageNet-trained CNNs prefer texture over shape, demonstrates that data augmentation largely drives this bias, and shows that naturalistic augmentations can promote shape-based classification and improve out-of-distribution performance.

ABSTRACT

Recent work has indicated that, unlike humans, ImageNet-trained CNNs tend to classify images by texture rather than by shape. How pervasive is this bias, and where does it come from? We find that, when trained on datasets of images with conflicting shape and texture, CNNs learn to classify by shape at least as easily as by texture. What factors, then, produce the texture bias in CNNs trained on ImageNet? Different unsupervised training objectives and different architectures have small but significant and largely independent effects on the level of texture bias. However, all objectives and architectures still lead to models that make texture-based classification decisions a majority of the time, even if shape information is decodable from their hidden representations. The effect of data augmentation is much larger. By taking less aggressive random crops at training time and applying simple, naturalistic augmentation (color distortion, noise, and blur), we train models that classify ambiguous images by shape a majority of the time, and outperform baselines on out-of-distribution test sets. Our results indicate that apparent differences in the way humans and ImageNet-trained CNNs process images may arise not primarily from differences in their internal workings, but from differences in the data that they see.

Motivation & Objective

• Assess whether texture bias in ImageNet-trained CNNs is intrinsic to architectures/training procedures or primarily due to training data.
• Quantify how different data augmentations, training objectives, and architectures affect texture bias.
• Identify practical augmentation strategies that reduce texture bias and improve shape-based classification on diverse distributions.
• Examine whether shape information is present in hidden representations even when decisions rely on texture.

Proposed method

• Train CNNs on ambiguous shape-texture datasets (GST, Navon, ImageNet-C) to compare shape vs. texture classification performance.
• Evaluate how training with different data augmentations (random-crop vs center-crop, color distortion, blur, noise, Sobel filtering) influences texture bias.
• Test various training objectives (supervised vs self-supervised: Rotation, Exemplar, BigBiGAN, SimCLR) and base architectures (AlexNet, ResNet-50).
• Measure shape bias using GST stimuli and assess how much shape/texture information is decodable from hidden layers by linear classifiers.
• Analyze the relationship between ImageNet accuracy and observed shape/texture biases across models.

Experimental results

Research questions

• RQ1 Is texture bias in ImageNet-trained CNNs primarily caused by inductive biases or by the training data itself?
• RQ2 How do data augmentation, training objectives, and architectures influence texture bias?
• RQ3 Can naturalistic augmentations reduce texture bias and improve shape-based classification on out-of-distribution data?
• RQ4 Is shape information recoverable from hidden representations even when models favor texture for classification?

Key findings

• CNNs trained on ambiguous datasets can learn to classify by shape as easily as by texture, indicating the bias is not solely an architectural property.
• Random-crop augmentation increases texture bias, while center-crop and naturalistic augmentations (color distortion, blur, noise, Sobel filtering) reduce texture bias.
• Augmentations that reduce texture bias have additive effects and can yield shape-biased models even without non-naturalistic style transfer techniques.
• Self-supervised learning objectives influence texture bias, but augmentation often plays a larger role; some objectives (e.g., Rotation) reduce texture bias relative to supervised baselines.
• Architectures with higher ImageNet accuracy tend to exhibit lower texture bias, though neural architectures designed to match human vision (or using self-attention) do not consistently differ in texture bias from standard CNNs.
• Shape information is decodable from final layers in texture-biased models, and in some cases decodability of shape exceeds that of texture in early layers, suggesting information loss occurs in later layers.