[Paper Review] Model Patching: Closing the Subgroup Performance Gap with Data Augmentation

Karan Goel, Albert Gu+2 more|arXiv (Cornell University)|Aug 15, 2020

Topic Modeling97 references45 citations

TL;DR

A two-stage framework called model patching (CAMEL) learns inter-subgroup transformations and uses CycleGAN-based augmentations with a subgroup consistency regularizer to improve subgroup robustness, reducing the subgroup performance gap and robust errors across benchmarks.

ABSTRACT

Classifiers in machine learning are often brittle when deployed. Particularly concerning are models with inconsistent performance on specific subgroups of a class, e.g., exhibiting disparities in skin cancer classification in the presence or absence of a spurious bandage. To mitigate these performance differences, we introduce model patching, a two-stage framework for improving robustness that encourages the model to be invariant to subgroup differences, and focus on class information shared by subgroups. Model patching first models subgroup features within a class and learns semantic transformations between them, and then trains a classifier with data augmentations that deliberately manipulate subgroup features. We instantiate model patching with CAMEL, which (1) uses a CycleGAN to learn the intra-class, inter-subgroup augmentations, and (2) balances subgroup performance using a theoretically-motivated subgroup consistency regularizer, accompanied by a new robust objective. We demonstrate CAMEL's effectiveness on 3 benchmark datasets, with reductions in robust error of up to 33% relative to the best baseline. Lastly, CAMEL successfully patches a model that fails due to spurious features on a real-world skin cancer dataset.

Motivation & Objective

• Motivate and address brittle classifier performance across subgroups within a class.
• Learn inter-subgroup transformations that preserve class labels.
• Train a classifier with data augmentations that manipulate subgroup features to improve invariance.
• Provide theoretical guarantees linking subgroup invariance to mutual information bounds.
• Empirically validate improvements on multiple benchmark datasets and a real-world skin cancer dataset.

Proposed method

• Stage 1 learns inter-subgroup transformations with CycleGANs to translate examples between subgroups within the same class.
• Stage 2 trains a classifier on original and augmented data using a subgroup robust objective and a subgroup consistency regularizer.
• The subgroup robust objective targets class-conditional worst-case subgroup performance (SGDRO).
• The subgroup consistency regularizer enforces invariance to subgroup features by aligning predictions across subgroup-augmented variants.
• The total objective combines SGDRO with a regularization term: L_CAMEL = L_SGDRO + lambda * L_c.

Experimental results

Research questions

• RQ1 Can learned inter-subgroup transformations produce meaningful, class-preserving augmentations between subgroups?
• RQ2 Does training with subgroup-focused augmentations and consistency regularization improve robustness and reduce subgroup gaps across datasets?
• RQ3 What are the theoretical guarantees connecting the augmentation and regularization to subgroup invariance?
• RQ4 How does CAMEL compare to ERM and robust baselines on benchmark datasets and real-world skin cancer data?

Key findings

• CAMEL achieves higher aggregate and robust accuracy while reducing the subgroup performance gap across benchmarks.
• On MNIST-Correlation, CAMEL reduces the subgroup gap to 0.96% with robust accuracy of 97.89%.
• On CelebA and Waterbirds, CAMEL consistently improves over baselines, reducing the subgroup gap by up to 10.22 percentage points on Waterbirds.
• On ISIC skin cancer data, CAMEL increases robust accuracy by 11.7% and improves malignant class accuracy from 65.59% to 78.86% (approximate in-text report).
• Ablations show learned augmentations + strong consistency regularization yield the best robustness, while heuristic augmentations or weaker consistency can hurt performance.
• MI estimates indicate CAMEL induces subgroup-invariant representations, with the MI estimate dropping to 0.02 in MNIST-Correlation (lower is better).