[Paper Review] Robust and Generalizable Visual Representation Learning via Random Convolutions

Zhenlin Xu, Deyi Liu+3 more|arXiv (Cornell University)|Jul 25, 2020

Domain Adaptation and Few-Shot LearningComputer Science48 references85 citations

TL;DR

The paper introduces random convolutions as a data augmentation to learn robust visual representations, improving domain generalization and sketch-image tasks by creating many texture-variant domains and combining them with originals.

ABSTRACT

While successful for various computer vision tasks, deep neural networks have shown to be vulnerable to texture style shifts and small perturbations to which humans are robust. In this work, we show that the robustness of neural networks can be greatly improved through the use of random convolutions as data augmentation. Random convolutions are approximately shape-preserving and may distort local textures. Intuitively, randomized convolutions create an infinite number of new domains with similar global shapes but random local textures. Therefore, we explore using outputs of multi-scale random convolutions as new images or mixing them with the original images during training. When applying a network trained with our approach to unseen domains, our method consistently improves the performance on domain generalization benchmarks and is scalable to ImageNet. In particular, in the challenging scenario of generalizing to the sketch domain in PACS and to ImageNet-Sketch, our method outperforms state-of-art methods by a large margin. More interestingly, our method can benefit downstream tasks by providing a more robust pretrained visual representation.

Motivation & Objective

• Address robustness to texture/style shifts and domain shift in visual representations.
• Develop a data augmentation strategy using random convolutions to promote global shape over local texture information.
• Investigate multi-scale random convolutions and mixing variants to produce diverse training views.
• Assess whether a more robust pretrained model improves downstream finetuning robustness.

Proposed method

• Apply random convolutions as a data augmentation technique to produce texture-randomized, approximately shape-preserving images.
• Use outputs of multi-scale random convolutions as new training inputs or mix them with original images.
• Train networks with augmented data to encourage invariance to local textures and domain shifts.
• Evaluate robustness and generalization on domain generalization benchmarks, including unseen domains.
• Test scalability by applying the approach to large-scale datasets like ImageNet.
• Provide open-source code for replication.

Experimental results

Research questions

• RQ1 Can random convolutions improve robustness to domain shifts and small perturbations in visual representations?
• RQ2 Does mixing random-convolution outputs with original images enhance generalization to unseen domains?
• RQ3 Is the approach scalable to large-scale datasets such as ImageNet and beneficial for downstream tasks via robust pretrained representations?
• RQ4 How does the method perform on challenging domain-shift benchmarks like PACS (Sketch domain) and ImageNet-Sketch?

Key findings

• Random convolutions improve robustness to domain shifts and small perturbations.
• Multi-scale random convolutions provide diverse texture variations that preserve global shapes.
• The method consistently improves performance on unseen domains in domain generalization benchmarks.
• It outperforms state-of-the-art methods on challenging settings like sketch domains (PACS) and ImageNet-Sketch.
• The approach is scalable to ImageNet and can yield more robust pretrained representations that benefit downstream tasks.