[Paper Review] Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Sets

Brandon M. Smith, Miguel Farinha+4 more|arXiv (Cornell University)|May 24, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes a synthetic data pipeline to debias vision-language datasets by generating gender-balanced contrast sets from COCO, where only the subject's gender is edited while preserving background context. Using a KNN-based filtering method to ensure image quality, the authors demonstrate that bias metrics like Bias@K are skewed by spurious correlations in real datasets, and show that balanced synthetic data leads to more reliable bias evaluation in CLIP-based models.

ABSTRACT

Vision-language models are growing in popularity and public visibility to generate, edit, and caption images at scale; but their outputs can perpetuate and amplify societal biases learned during pre-training on uncurated image-text pairs from the internet. Although debiasing methods have been proposed, we argue that these measurements of model bias lack validity due to dataset bias. We demonstrate there are spurious correlations in COCO Captions, the most commonly used dataset for evaluating bias, between background context and the gender of people in-situ. This is problematic because commonly-used bias metrics (such as Bias@K) rely on per-gender base rates. To address this issue, we propose a novel dataset debiasing pipeline to augment the COCO dataset with synthetic, gender-balanced contrast sets, where only the gender of the subject is edited and the background is fixed. However, existing image editing methods have limitations and sometimes produce low-quality images; so, we introduce a method to automatically filter the generated images based on their similarity to real images. Using our balanced synthetic contrast sets, we benchmark bias in multiple CLIP-based models, demonstrating how metrics are skewed by imbalance in the original COCO images. Our results indicate that the proposed approach improves the validity of the evaluation, ultimately contributing to more realistic understanding of bias in vision-language models.

Motivation & Objective

• To identify and address spurious correlations between gender and background context in the COCO dataset that distort bias evaluation in vision-language models.
• To develop a scalable, automated pipeline for generating high-quality, gender-balanced synthetic contrast sets from real images, preserving scene context while editing only the subject's gender.
• To improve the validity of bias metrics like Bias@K by eliminating dataset-level imbalances that confound model bias assessment.
• To demonstrate that observed model bias in CLIP-based models is often attributable to dataset artifacts rather than intrinsic model bias.
• To initiate a discussion on the ethical implications of synthetic data generation, including risks of bias stacking and oversimplification of gender identity.

Proposed method

• The method generates synthetic image pairs from COCO using a controllable image editing model, where each image is edited to depict the same scene with a different gendered subject.
• Only the person's bounding box is modified; the background and scene context remain unchanged to isolate gender as the sole variable.
• A KNN-based filtering pipeline compares synthetic images to real images in feature space to reject low-quality or unrealistic edits.
• The filtered synthetic dataset, named GenSynth, is used to re-evaluate bias metrics on multiple CLIP-based models.
• The approach enables controlled evaluation of gender bias by ensuring equal representation of genders across fixed contexts.
• The pipeline avoids manual curation and enables large-scale debiasing of vision-language datasets.

Figure 1 : t-SNE clusters ( $M=20$ ) of gender-neutralised caption embeddings. Each cluster is manually assigned a name, then coloured and labelled according to its male over-representation factor. The male over-representation factor is the difference between the percentage of male images in the par

Experimental results

Research questions

• RQ1 To what extent do spurious correlations between gender and scene context in COCO distort standard bias metrics like Bias@K?
• RQ2 Can synthetic contrast sets with balanced gender representation across fixed backgrounds improve the validity of bias evaluation in vision-language models?
• RQ3 How do bias metrics change when evaluated on a balanced synthetic dataset compared to the original, imbalanced COCO dataset?
• RQ4 To what extent is observed model bias in CLIP-based models due to dataset bias rather than inherent model associations?
• RQ5 What are the risks and limitations of using synthetic data for debiasing, particularly regarding distribution shift and bias propagation?

Key findings

• The COCO dataset exhibits strong spurious correlations between gender and background context, which invalidates standard bias metrics like Bias@K.
• The proposed GenSynth synthetic contrast set successfully removes gender-by-context bias, enabling more reliable evaluation of model fairness.
• Bias@K values on the original COCO dataset are significantly skewed due to imbalanced gender representation in scene contexts.
• When evaluated on the balanced GenSynth dataset, CLIP-based models show similar bias scores across genders, indicating that observed disparities were largely due to dataset artifacts.
• The filtering pipeline based on KNN similarity to real images effectively maintains image quality and reduces distribution shift in synthetic data.
• Despite limitations, the study shows that synthetic contrast sets can serve as a valid and scalable alternative for debiasing vision-language evaluation.

Figure 2 : An overview of our pipeline for dataset debiasing across a target attribute, in this case gender, ensuring equal demographic representation. A source image containing a person is given as input to InstructPix2Pix along with instructions to synthesise each attribute label. The resulting ed