[Paper Review] Learning Visual Representations for Transfer Learning by Suppressing Texture

Shlok Mishra, Anshul Shah+6 more|arXiv (Cornell University)|Nov 3, 2020

Domain Adaptation and Few-Shot Learning41 references4 citations

TL;DR

This paper proposes suppressing texture in ImageNet images using anisotropic diffusion as a data augmentation technique to encourage CNNs to learn higher-level, shape-based representations instead of relying on texture biases. The method improves transfer learning performance across 12 diverse datasets, with up to 11.49% accuracy gains on Sketch-ImageNet and 10.41% on Synthetic-DTD, demonstrating reduced reliance on low-level cues.

ABSTRACT

Recent literature has shown that features obtained from supervised training of CNNs may over-emphasize texture rather than encoding high-level information. In self-supervised learning in particular, texture as a low-level cue may provide shortcuts that prevent the network from learning higher level representations. To address these problems we propose to use classic methods based on anisotropic diffusion to augment training using images with suppressed texture. This simple method helps retain important edge information and suppress texture at the same time. We empirically show that our method achieves state-of-the-art results on object detection and image classification with eight diverse datasets in either supervised or self-supervised learning tasks such as MoCoV2 and Jigsaw. Our method is particularly effective for transfer learning tasks and we observed improved performance on five standard transfer learning datasets. The large improvements (up to 11.49\%) on the Sketch-ImageNet dataset, DTD dataset and additional visual analyses with saliency maps suggest that our approach helps in learning better representations that better transfer.

Motivation & Objective

• To address the problem of CNNs over-relying on texture and low-level features, which hinders generalization across domains.
• To improve transfer learning performance by encouraging models to focus on high-level, shape-based representations instead of texture.
• To develop a simple, effective data augmentation method that suppresses texture while preserving structural edges.
• To outperform existing methods like Gaussian blur and Stylized-ImageNet in both supervised and self-supervised learning settings.

Proposed method

• Apply anisotropic diffusion to ImageNet images to suppress texture while preserving edges and structural boundaries.
• Use the Perona-Malik and robust anisotropic diffusion frameworks to generate texture-suppressed images for data augmentation.
• Integrate the augmented dataset into pretraining for both supervised and self-supervised learning (e.g., MoCoV2, Dense-CL).
• Compare with baseline methods including Gaussian blur and Stylized-ImageNet, using the same training protocols.
• Train models on the augmented dataset and evaluate on downstream transfer learning tasks across diverse benchmarks.
• Use Pix2Pix to learn a conditional image-to-image translation for texture suppression, achieving better performance than style transfer or blurring.

Experimental results

Research questions

• RQ1 Can suppressing texture in ImageNet images improve the generalization of pre-trained CNNs on downstream transfer learning tasks?
• RQ2 Does anisotropic diffusion-based data augmentation lead to representations less dependent on texture and more focused on shape and structure?
• RQ3 How does texture suppression compare to Gaussian blurring and style transfer in terms of performance and robustness across diverse datasets?
• RQ4 To what extent does the proposed method reduce reliance on high-frequency texture cues in CNNs?
• RQ5 Can the method improve performance on datasets where texture is a spurious cue, such as Synthetic-DTD and GST?

Key findings

• The proposed method achieved a 11.49% accuracy improvement on Sketch-ImageNet, indicating stronger learning of high-level, shape-based representations.
• On the Synthetic-DTD dataset, the model showed a 10.41% performance boost, confirming reduced reliance on texture and improved generalization.
• The method outperformed both Gaussian blur and Stylized-ImageNet across multiple transfer learning benchmarks, including VOC, DTD, Cars, and Aircraft.
• Qualitative analysis of saliency maps confirmed that the model is less dependent on high-frequency texture information and more focused on structural features.
• The best performance was achieved using the simple Perona-Malik anisotropic diffusion method, while increasing iterations degraded performance due to distribution shift.
• On the GST dataset, the model improved by 1.02% in shape-based classification, demonstrating reduced texture bias.