[Paper Review] BiaSwap: Removing dataset bias with bias-tailored swapping augmentation

Eungyeup Kim, Jihyeon Lee+1 more|arXiv (Cornell University)|Aug 23, 2021

Domain Adaptation and Few-Shot Learning29 references4 citations

TL;DR

BiaSwap proposes a bias-tailored image augmentation method that removes dataset bias without requiring prior knowledge of bias types. By leveraging class activation maps (CAMs) and a style-transfer module, it swaps bias-relevant attributes from 'bias-contrary' images onto 'bias-guiding' images, generating realistic, debiased training samples that improve generalization on both biased and unbiased data.

ABSTRACT

Deep neural networks often make decisions based on the spurious correlations inherent in the dataset, failing to generalize in an unbiased data distribution. Although previous approaches pre-define the type of dataset bias to prevent the network from learning it, recognizing the bias type in the real dataset is often prohibitive. This paper proposes a novel bias-tailored augmentation-based approach, BiaSwap, for learning debiased representation without requiring supervision on the bias type. Assuming that the bias corresponds to the easy-to-learn attributes, we sort the training images based on how much a biased classifier can exploits them as shortcut and divide them into bias-guiding and bias-contrary samples in an unsupervised manner. Afterwards, we integrate the style-transferring module of the image translation model with the class activation maps of such biased classifier, which enables to primarily transfer the bias attributes learned by the classifier. Therefore, given the pair of bias-guiding and bias-contrary, BiaSwap generates the bias-swapped image which contains the bias attributes from the bias-contrary images, while preserving bias-irrelevant ones in the bias-guiding images. Given such augmented images, BiaSwap demonstrates the superiority in debiasing against the existing baselines over both synthetic and real-world datasets. Even without careful supervision on the bias, BiaSwap achieves a remarkable performance on both unbiased and bias-guiding samples, implying the improved generalization capability of the model.

Motivation & Objective

• Address dataset bias in deep learning without requiring explicit supervision on bias types.
• Improve model generalization by reducing reliance on spurious correlations in training data.
• Maintain high performance on both bias-guiding (biased) and bias-contrary (unbiased) samples.
• Develop a realistic augmentation method that transfers only bias-relevant attributes while preserving semantic content.
• Enable unsupervised identification of bias-guiding and bias-contrary samples using the assumption that bias is easy-to-learn.

Proposed method

• Use class activation maps (CAMs) from a biased classifier to identify regions where spurious correlations are exploited.
• Sort training images into 'bias-guiding' and 'bias-contrary' sets based on how much a biased classifier exploits easy-to-learn attributes.
• Integrate CAMs with a style-transfer-based image translation framework to transfer only bias-relevant attributes from bias-contrary to bias-guiding images.
• Apply a patch-based discriminator guided by CAMs to ensure localized, attribute-specific style transfer.
• Use GCE (Generalized Cross-Entropy) loss to stabilize training and improve alignment between source and target attributes.
• Generate 'bias-swapped' images that preserve non-bias attributes from the source while adopting bias attributes from the target.

Experimental results

Research questions

• RQ1 Can we effectively detect and mitigate dataset bias without prior knowledge of the bias type?
• RQ2 How can we generate realistic, debiased training samples by transferring only bias-relevant attributes?
• RQ3 Does unsupervised identification of bias-guiding and bias-contrary samples improve model generalization?
• RQ4 Can bias-swapped images outperform existing augmentation-based debiasing methods in both unbiased and biased test scenarios?
• RQ5 How does the realism of augmented images impact debiasing performance compared to methods producing unnatural artifacts?

Key findings

• BiaSwap achieves state-of-the-art unbiased test accuracy on Colored MNIST (83.74%), outperforming SIN (40.79%) and vanilla models (48.76%).
• On the real-world BAR dataset, BiaSwap achieves 52.44% unbiased accuracy, surpassing SIN (50.51%) and the vanilla baseline (49.59%).
• On bFFHQ, BiaSwap attains 78.98% unbiased accuracy, significantly exceeding SIN (69.86%) and the vanilla model (74.86%).
• BiaSwap maintains strong performance on bias-guiding samples, indicating improved generalization without sacrificing accuracy on biased data.
• Visualizations confirm that BiaSwap successfully transfers only bias-relevant attributes (e.g., gender, color, texture) while preserving semantic structure.
• The method generates more realistic augmented images than SIN, which often produces unnatural artifacts like distorted shapes or inconsistent textures.