[Paper Review] Self-Challenging Improves Cross-Domain Generalization

Zeyi Huang, Haohan Wang+2 more|arXiv (Cornell University)|Jul 5, 2020

Domain Adaptation and Few-Shot LearningComputer Science26 references43 citations

TL;DR

Introduces Representation Self-Challenging (RSC), a gradient-based training heuristic that mutes dominant features during training to improve CNN generalization across unseen domains without extra parameters.

ABSTRACT

Convolutional Neural Networks (CNN) conduct image classification by activating dominant features that correlated with labels. When the training and testing data are under similar distributions, their dominant features are similar, which usually facilitates decent performance on the testing data. The performance is nonetheless unmet when tested on samples from different distributions, leading to the challenges in cross-domain image classification. We introduce a simple training heuristic, Representation Self-Challenging (RSC), that significantly improves the generalization of CNN to the out-of-domain data. RSC iteratively challenges (discards) the dominant features activated on the training data, and forces the network to activate remaining features that correlates with labels. This process appears to activate feature representations applicable to out-of-domain data without prior knowledge of new domain and without learning extra network parameters. We present theoretical properties and conditions of RSC for improving cross-domain generalization. The experiments endorse the simple, effective and architecture-agnostic nature of our RSC method.

Motivation & Objective

• Motivate the need for cross-domain generalization beyond similar training/testing distributions.
• Propose a simple, domain-agnostic training heuristic (RSC) to enhance generalization.
• Provide theoretical analysis showing how RSC tightens generalization bounds.
• Empirically validate RSC across standard DG benchmarks and ablations.

Proposed method

• At each iteration, compute the gradient of the top classifier with respect to the feature representation z.
• Create a mask that zeroes out the top-p percentile of gradient magnitudes, effectively discarding the most predictive features.
• Mute the corresponding elements in z to form a perturbed representation {z}.
• Compute the softmax output using the perturbed z and backpropagate to update all network parameters.
• RSC requires no extra parameters and is architecture-agnostic; p is a hyperparameter controlling the discarded feature proportion.
• Extensions include spatial-wise and channel-wise RSC, and batch-based application strategies.

Experimental results

Research questions

• RQ1 Does RSC improve cross-domain generalization without domain information?
• RQ2 How does the dropping proportion p affect training dynamics and generalization?
• RQ3 Is gradient-based feature dropping more effective than activation-based or random dropout strategies?
• RQ4 Does combining spatial and channel-wise RSC further boost DG performance?
• RQ5 How does RSC scale with backbone architecture and dataset complexity?

Key findings

• RSC consistently improves cross-domain generalization across multiple DG benchmarks (e.g., PACS, VLCS, Office-Home, ImageNet-Sketch).
• On PACS, RSC yields substantial improvements over baselines, with reported gains like 4.5 on AlexNet, 5.2 on ResNet18, and 4.5 on ResNet50 over selected baselines.
• In ablations, Top-Gradient feature dropping outperforms Top-Activation and random dropping, and an optimal p is around one-third for several settings.
• Spatial-only RSC and Spatial+Channel RSC outperform baseline and several dropout variants, demonstrating additional gains when leveraging convolutional structure.
• RSC also narrows performance gaps between network sizes on ImageNet, suggesting efficiency in achieving stronger generalization without larger models.