[Paper Review] Frustratingly Simple Domain Generalization via Image Stylization

Nathan Somavarapu, Chih‐Yao Ma+1 more|arXiv (Cornell University)|Jun 19, 2020

Image Retrieval and Classification Techniques40 references53 citations

TL;DR

A simple in-domain stylization augmentation (and inter-source stylization) improves domain generalization by biasing models toward shape over texture, achieving competitive or state-of-the-art results on DG benchmarks without external data sources.

ABSTRACT

Convolutional Neural Networks (CNNs) show impressive performance in the standard classification setting where training and testing data are drawn i.i.d. from a given domain. However, CNNs do not readily generalize to new domains with different statistics, a setting that is simple for humans. In this work, we address the Domain Generalization problem, where the classifier must generalize to an unknown target domain. Inspired by recent works that have shown a difference in biases between CNNs and humans, we demonstrate an extremely simple yet effective method, namely correcting this bias by augmenting the dataset with stylized images. In contrast with existing stylization works, which use external data sources such as art, we further introduce a method that is entirely in-domain using no such extra sources of data. We provide a detailed analysis as to the mechanism by which the method works, verifying our claim that it changes the shape/texture bias, and demonstrate results surpassing or comparable to the state of the arts that utilize much more complex methods.

Motivation & Objective

• Motivate robust generalization to unseen domains by addressing model biases that differ from humans.
• Propose a lightweight data augmentation that alters texture while preserving content to shift model reliance toward shape.
• Show that in-domain and inter-source stylization can match or surpass state-of-the-art DG methods on standard benchmarks.
• Analyze how stylization changes shape/texture bias and its impact on generalization.

Proposed method

• Define probabilistic stylization S_{Q,p}(x) that replaces an image with a stylized version with probability p using AdaIN, with style set Q drawn from sources like painter datasets or other source domains.
• Apply stylization during training in an end-to-end optimization of the classification loss: min_theta E_{x,y ~ D_S}[ L_c( f(S_{Q,p}(x); theta), y) ].
• Introduce Inter-Source Stylization where Q consists of other source domains (or intra-source where Q is the current domain) to leverage in-domain variability without external data.
• Demonstrate that stylization tends to increase shape information while not dramatically reducing texture information, leading to better cross-domain performance.
• Experiment with DG benchmarks PACS, VLCS, and Office-Home using ResNet-18 and AlexNet backbones.
• Provide analyses including cue-conflict datasets to measure shape bias, shape accuracy, and texture accuracy.

Experimental results

Research questions

• RQ1 Does simple stylization augmentation improve domain generalization without extra data sources?
• RQ2 How does stylization affect shape vs. texture bias in CNNs in DG settings?
• RQ3 Is inter-source stylization as effective as stylization with external painting data for DG?
• RQ4 Which source domains contribute most to the benefits of inter-source stylization?
• RQ5 Are the gains consistent across different backbone architectures and datasets?

Key findings

• Stylization-based augmentation yields competitive to state-of-the-art DG results on PACS, VLCS, and Office-Home benchmarks.
• Inter-Source Stylization (no external painting data) provides comparable improvements to painting-based stylization, indicating sufficient cross-source variability.
• Stylization increases shape bias and shape accuracy with texture accuracy remaining largely unchanged, correlating with better cross-domain generalization.
• Benefits are observed across backbones (ResNet-18 and AlexNet) and across multiple target domains.
• Using more sources for inter-source stylization generally helps, with diminishing returns beyond a mixture of multiple sources.