[Paper Review] SelecMix: Debiased Learning by Contradicting-pair Sampling

Inwoo Hwang, Sang-Jun Lee+5 more|arXiv (Cornell University)|Nov 4, 2022

Machine Learning and Data Classification4 citations

TL;DR

SelecMix proposes a debiased learning method that uses mixup on contradicting pairs—examples with the same label but dissimilar biased features, or different labels but similar biased features—to generate bias-conflicting training samples. By training an auxiliary contrastive model to emphasize easy-to-learn biased features, SelecMix identifies these pairs without requiring explicit bias labels or disentangled representations, achieving state-of-the-art performance on benchmark datasets, especially under label noise and low-bias-conflict data regimes.

ABSTRACT

Neural networks trained with ERM (empirical risk minimization) sometimes learn unintended decision rules, in particular when their training data is biased, i.e., when training labels are strongly correlated with undesirable features. To prevent a network from learning such features, recent methods augment training data such that examples displaying spurious correlations (i.e., bias-aligned examples) become a minority, whereas the other, bias-conflicting examples become prevalent. However, these approaches are sometimes difficult to train and scale to real-world data because they rely on generative models or disentangled representations. We propose an alternative based on mixup, a popular augmentation that creates convex combinations of training examples. Our method, coined SelecMix, applies mixup to contradicting pairs of examples, defined as showing either (i) the same label but dissimilar biased features, or (ii) different labels but similar biased features. Identifying such pairs requires comparing examples with respect to unknown biased features. For this, we utilize an auxiliary contrastive model with the popular heuristic that biased features are learned preferentially during training. Experiments on standard benchmarks demonstrate the effectiveness of the method, in particular when label noise complicates the identification of bias-conflicting examples.

Motivation & Objective

• To address the problem of deep neural networks learning spurious correlations from biased training data, especially when robust features are underrepresented.
• To develop a data augmentation method that generates effective bias-conflicting examples without relying on complex generative models or disentangled representations.
• To improve model generalization under distribution shift and label noise by selectively augmenting data using contradicting pairs.
• To validate the effectiveness of the method on standard debiasing benchmarks, particularly when bias-conflicting examples are scarce.

Proposed method

• SelecMix applies mixup to contradicting pairs of examples: (i) same label but dissimilar biased features, or (ii) different labels but similar biased features.
• An auxiliary contrastive model is trained with a generalized supervised contrastive (GSC) loss to amplify reliance on biased features, enabling identification of such pairs.
• The auxiliary model's embedding space is used to measure similarity of biased features, allowing selection of contradicting pairs without explicit bias labels.
• Mixup is applied selectively to these pairs, with a symmetric weighting strategy where λ is set to min(λ, 1−λ) to favor the selected sample.
• The debiased model is trained jointly with the auxiliary model, using the augmented data to improve robustness.
• The method implicitly upweights existing bias-conflicting examples by frequently selecting them in mixup, enhancing their influence without explicit reweighting.

Experimental results

Research questions

• RQ1 Can mixup be effectively adapted to generate bias-conflicting examples without requiring explicit bias labels or disentangled representations?
• RQ2 How does the selective mixup strategy based on contradicting pairs compare to standard mixup and other data augmentation baselines in debiasing?
• RQ3 To what extent does the auxiliary contrastive model with GSC loss improve the identification of biased feature similarity for pair selection?
• RQ4 How robust is SelecMix to label noise, which complicates the detection of true bias-conflicting examples?
• RQ5 Does SelecMix maintain superior performance when bias-conflicting examples are rare in the training data?

Key findings

• SelecMix outperforms prior methods on standard debiasing benchmarks, achieving 70.80% accuracy on the Colored MNIST dataset with 0.5% bias ratio, compared to 58.07% for vanilla mixup.
• On the BFFHQ dataset with 0.5% bias ratio, SelecMix achieves 77.40% accuracy, significantly outperforming LISA (58.93%) and vanilla mixup (56.20%).
• In the presence of label noise, SelecMix maintains strong performance, demonstrating robustness where other methods degrade.
• Abllation studies confirm that both the selective mixup strategy and the GSC loss are essential, with the latter improving biased feature similarity measurement.
• SelecMix achieves 91.57% accuracy on Corrupted CIFAR-10 with 5% bias ratio, outperforming LISA (88.73%) and vanilla mixup (82.12%).
• The method shows consistent gains across all tested datasets and bias ratios, especially when bias-conflicting examples are scarce, indicating effective implicit upweighting.