[Paper Review] Adversarial Style Augmentation for Domain Generalization

Yabin Zhang, Bin Deng+3 more|arXiv (Cornell University)|Jan 30, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Adversarial Style Augmentation (ASA), a novel domain generalization method that enhances model robustness by adversarially learning optimal feature statistics perturbations (mean and standard deviation) beyond mini-batch constraints. By jointly optimizing perturbation direction and intensity via adversarial training and implementing it through a plug-and-play module called AdvStyle, the method achieves state-of-the-art performance, boosting ResNet-50 accuracy on PACS from 61.2% to 67.1% under single-source generalization.

ABSTRACT

It is well-known that the performance of well-trained deep neural networks may degrade significantly when they are applied to data with even slightly shifted distributions. Recent studies have shown that introducing certain perturbation on feature statistics (\eg, mean and standard deviation) during training can enhance the cross-domain generalization ability. Existing methods typically conduct such perturbation by utilizing the feature statistics within a mini-batch, limiting their representation capability. Inspired by the domain generalization objective, we introduce a novel Adversarial Style Augmentation (ASA) method, which explores broader style spaces by generating more effective statistics perturbation via adversarial training. Specifically, we first search for the most sensitive direction and intensity for statistics perturbation by maximizing the task loss. By updating the model against the adversarial statistics perturbation during training, we allow the model to explore the worst-case domain and hence improve its generalization performance. To facilitate the application of ASA, we design a simple yet effective module, namely AdvStyle, which instantiates the ASA method in a plug-and-play manner. We justify the efficacy of AdvStyle on tasks of cross-domain classification and instance retrieval. It achieves higher mean accuracy and lower performance fluctuation. Especially, our method significantly outperforms its competitors on the PACS dataset under the single source generalization setting, \eg, boosting the classification accuracy from 61.2\% to 67.1\% with a ResNet50 backbone. Our code will be available at \url{https://github.com/YBZh/AdvStyle}.

Motivation & Objective

• To address the limitation of existing domain generalization methods that restrict statistics perturbation to the current mini-batch, limiting style augmentation diversity.
• To explore a broader style space by learning more effective and robust statistics perturbations (mean and standard deviation) beyond batch-level statistics.
• To improve cross-domain generalization performance in vision tasks such as classification and instance retrieval by simulating worst-case domain shifts during training.
• To design a simple, end-to-end trainable module (AdvStyle) that enables plug-and-play integration into existing deep learning models.

Proposed method

• The method introduces Adversarial Style Augmentation (ASA), which models feature statistics (mean and standard deviation) as learnable parameters and optimizes them via adversarial training.
• It performs a two-player optimization: the model is trained to minimize task loss against the worst-case statistics perturbation, while the perturbation parameters are updated to maximize the task loss.
• The perturbation is generated by sampling from Gaussian distributions centered on the batch’s mean and standard deviation, with the standard deviations treated as learnable parameters.
• The AdvStyle module integrates this mechanism into a neural network via Gradient Reversal Layers (GRL), enabling end-to-end training and plug-and-play deployment.
• The perturbation direction and intensity are jointly optimized, allowing the model to explore diverse and challenging style shifts during training.
• The approach is implemented in a differentiable manner, enabling backpropagation through the perturbation generation process.

Experimental results

Research questions

• RQ1 Can adversarial learning of feature statistics perturbation improve domain generalization beyond the constraints of mini-batch statistics?
• RQ2 Is exploring the direction of statistics perturbation more effective than only adjusting its intensity for domain generalization?
• RQ3 How does the proposed AdvStyle module compare to existing style augmentation methods in terms of accuracy and robustness across different domains?
• RQ4 What is the impact of inserting AdvStyle at different layers in a deep network architecture on generalization performance?
• RQ5 How stable is the method across different hyperparameter settings, particularly the strength of perturbation (λ)?

Key findings

• The proposed AdvStyle module significantly improves generalization performance, achieving 67.1% top-1 accuracy on the PACS dataset under single-source generalization, a 5.9% absolute gain over the baseline ResNet-50.
• AdvStyle outperforms the closest baseline, DSU (Li et al., 2022), by 9.8 percentage points on PACS, demonstrating its superiority in style space exploration.
• Ablation studies show that inserting AdvStyle at multiple positions in the ResNet backbone consistently improves performance, with the best result achieved when applied at all six analyzed layers.
• The ablation variant AdvStyle-Direction-Only (64.9%) outperforms AdvStyle-Intensity-Only (61.6%), indicating that learning the perturbation direction is more effective than adjusting intensity alone.
• The method remains stable across a wide range of λ values (0.5 to 20), with λ = 5 found to yield consistently strong performance, confirming robustness to hyperparameter choice.
• Visualization via t-SNE and A-distance confirms that AdvStyle leads to greater domain alignment and reduced distribution divergence, validating its ability to expand the effective style space.