[Paper Review] Learning Robust Global Representations by Penalizing Local Predictive Power

Haohan Wang, Songwei Ge+2 more|arXiv (Cornell University)|May 29, 2019

Domain Adaptation and Few-Shot Learning59 references278 citations

TL;DR

The paper introduces Patch-wise Adversarial Regularization (PAR) to suppress local (patch-level) predictive signals in early CNN layers, pushing models to learn global representations that generalize better under domain shift without requiring target-domain data.

ABSTRACT

Despite their renowned predictive power on i.i.d. data, convolutional neural networks are known to rely more on high-frequency patterns that humans deem superficial than on low-frequency patterns that agree better with intuitions about what constitutes category membership. This paper proposes a method for training robust convolutional networks by penalizing the predictive power of the local representations learned by earlier layers. Intuitively, our networks are forced to discard predictive signals such as color and texture that can be gleaned from local receptive fields and to rely instead on the global structures of the image. Across a battery of synthetic and benchmark domain adaptation tasks, our method confers improved generalization out of the domain. Also, to evaluate cross-domain transfer, we introduce ImageNet-Sketch, a new dataset consisting of sketch-like images, that matches the ImageNet classification validation set in categories and scale.

Motivation & Objective

• Motivate robust image classification that relies on global rather than local cues to improve out-of-domain generalization.
• Propose a training objective that penalizes local predictive power in early layers while preserving final-layer accuracy.
• Explore practical variants and training heuristics of the method across synthetic and real-world domain adaptation/generalization tasks.
• Introduce ImageNet-Sketch as a large-scale out-of-domain benchmark to evaluate robustness on sketch-like images.

Proposed method

• Define a patch-wise classifier h that operates on local features g(X;δ) at each spatial location.
• Formulate a minimax regularization that penalizes local predictive power: for each location, train h to predict y from local patch representations while training g to hinder such predictions (Eq. 2).
• Combine PAR objective with standard loss: minimize E[l(f(g(X;δ);θ),y)] − (λ/m′n′) Σ l(h(g(X;δ)_{i,j};φ),y) over δ,θ while minimizing the local-prediction loss over φ.
• Implement h as a 1×1 convolution (shared across locations) for efficiency; variants include stronger local classifiers (PAR M), larger patches (PAR B), and using higher layers (PAR H).
• Optionally apply a training heuristic: pretrain conventionally then fine-tune with PAR.
• Present extensions and practical training details to compare with domain-adversarial and other domain-generalization methods.

Experimental results

Research questions

• RQ1 Does penalizing local predictive power in early layers improve robustness to domain shifts without access to target-domain data?
• RQ2 How does PAR compare to existing domain adaptation/generalization methods under various perturbations and dataset shifts?
• RQ3 What are the effects of different PAR variants (vanilla, MLP discriminator, larger patches, higher-layer regularization) on in-domain vs out-of-domain performance?
• RQ4 Can a large-scale, sketch-based out-of-domain benchmark (ImageNet-Sketch) reveal robustness advantages of PAR beyond standard benchmarks?

Key findings

• PAR and its variants improve out-of-domain/generalization performance across MNIST perturbations, CIFAR-10 perturbations, and PACS, often outperforming DANN, HEX, and InfoDrop baselines.
• PAR achieves notable gains on theSketch-domain of PACS, highlighting its strength when color/local cues do not transfer.
• On CIFAR-10 with perturbations, PAR yields the best or near-best accuracy across greyscale, negative color, random kernel, and radial kernel perturbations, with PAR M/B/H variations performing differently per perturbation.
• ImageNet-Sketch experiments show PAR modestly improves top-1/top-5 accuracy over AlexNet baseline when evaluated on sketch-like images, demonstrating improved cross-domain generalization.
• In some in-domain settings where local patterns are truly predictive, PAR can slightly reduce in-domain performance; overall, vanilla PAR offers robust improvements across many scenarios.