[Paper Review] Improving Fairness in Image Classification via Sketching

Ruichen Yao, Ziteng Cui+2 more|arXiv (Cornell University)|Oct 31, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a novel pre-processing approach to improve fairness in image classification by converting input images into sketches, which inherently filter out bias-inducing features like skin color and texture while preserving semantic information. By combining a CLIP-guided sketching model with a fairness loss, the method achieves state-of-the-art fairness improvements across multiple datasets with minimal accuracy drop.

ABSTRACT

Fairness is a fundamental requirement for trustworthy and human-centered Artificial Intelligence (AI) system. However, deep neural networks (DNNs) tend to make unfair predictions when the training data are collected from different sub-populations with different attributes (i.e. color, sex, age), leading to biased DNN predictions. We notice that such a troubling phenomenon is often caused by data itself, which means that bias information is encoded to the DNN along with the useful information (i.e. class information, semantic information). Therefore, we propose to use sketching to handle this phenomenon. Without losing the utility of data, we explore the image-to-sketching methods that can maintain useful semantic information for the target classification while filtering out the useless bias information. In addition, we design a fair loss to further improve the model fairness. We evaluate our method through extensive experiments on both general scene dataset and medical scene dataset. Our results show that the desired image-to-sketching method improves model fairness and achieves satisfactory results among state-of-the-art.

Motivation & Objective

• To address fairness issues in deep neural networks caused by biased training data, particularly in attributes like skin color, gender, and age.
• To explore whether image sketching can filter out bias-inducing visual features (e.g., color, texture) without sacrificing semantic information for classification.
• To develop a simple yet effective pre-processing pipeline that improves fairness without modifying the model architecture or inference process.
• To evaluate the effectiveness of sketching combined with a fairness loss on both general and medical image datasets.
• To demonstrate that sketching can achieve state-of-the-art fairness performance with minimal accuracy degradation.

Proposed method

• The method uses a CLIP-guided image-to-sketching model to convert input images into sketches, preserving semantic and class-relevant features while discarding color and texture information that contribute to bias.
• The sketch images are then fed into a standard classification model, trained with a cross-entropy classification loss to maintain accuracy.
• A fairness loss term is introduced to further minimize bias, with the total loss being a weighted sum of classification loss and fairness loss.
• The fairness loss is designed to reduce disparities in model predictions across different sensitive attributes (e.g., gender, skin tone, age).
• The sketching pipeline is applied offline during training, making it a pre-processing technique that does not require changes to the inference phase.
• The method is evaluated using standard fairness metrics: Statistical Parity Difference (SPD), Equal Opportunity Difference (EOD), Difference in Equalized Odds (DEO), and Average Odds Difference (AOD).

Experimental results

Research questions

• RQ1 Can image sketching effectively reduce bias in image classification models by filtering out attribute-specific visual cues like skin color and texture?
• RQ2 Does sketching preserve sufficient semantic and class-relevant information to maintain high classification accuracy while reducing bias?
• RQ3 How does combining sketching with a fairness loss compare to existing fairness methods in terms of fairness metrics and accuracy?
• RQ4 Can this approach generalize across diverse datasets, including medical imaging (e.g., ISIC) and general scene datasets (e.g., CelebA)?
• RQ5 Does the addition of a fairness loss further improve fairness without significantly degrading model accuracy?

Key findings

• On the CelebA dataset with gender as a sensitive attribute, the method reduced SPD from 0.0707 (original) to 0.0333 with sketching and fairness loss, showing significant fairness improvement.
• For skin color bias on CelebA, the method reduced SPD from 0.0707 to 0.0333 and DEO from 0.0916 to 0.0564, demonstrating strong bias mitigation.
• On the ISIC dataset, the method reduced SPD from 0.246 (original) to 0.223 for age-related bias and from 0.134 to 0.144 for skin tone bias, with improved fairness in EOD and AOD metrics.
• The method achieved competitive or superior fairness performance compared to a SOTA fairness method (fair training + FAA), particularly in SPD reduction for gender bias.
• Despite a slight accuracy drop when using the fairness loss, the overall performance remained strong, with minimal degradation in F1-score and precision across all sensitive attributes.
• The results confirm that sketching is an effective pre-processing technique for fairness, especially when combined with a fairness loss, across diverse image classification tasks.