[Paper Review] Can We Achieve Fairness Using Semi-Supervised Learning?

Joymallya Chakraborty, Huy Tu+2 more|arXiv (Cornell University)|Nov 3, 2021

Adversarial Robustness in Machine Learning53 references4 citations

TL;DR

This paper proposes Fair-SSL, a semi-supervised learning framework that uses pseudo-labeling and data augmentation to achieve fairness in machine learning models with only 10% labeled data. By generating balanced, pseudo-labeled training data, Fair-SSL achieves performance comparable to state-of-the-art bias mitigation methods while significantly reducing reliance on costly labeled data.

ABSTRACT

Ethical bias in machine learning models has become a matter of concern in the software engineering community. Most of the prior software engineering works concentrated on finding ethical bias in models rather than fixing it. After finding bias, the next step is mitigation. Prior researchers mainly tried to use supervised approaches to achieve fairness. However, in the real world, getting data with trustworthy ground truth is challenging and also ground truth can contain human bias. Semi-supervised learning is a machine learning technique where, incrementally, labeled data is used to generate pseudo-labels for the rest of the data (and then all that data is used for model training). In this work, we apply four popular semi-supervised techniques as pseudo-labelers to create fair classification models. Our framework, Fair-SSL, takes a very small amount (10%) of labeled data as input and generates pseudo-labels for the unlabeled data. We then synthetically generate new data points to balance the training data based on class and protected attribute as proposed by Chakraborty et al. in FSE 2021. Finally, the classification model is trained on the balanced pseudo-labeled data and validated on test data. After experimenting on ten datasets and three learners, we find that Fair-SSL achieves similar performance as three state-of-the-art bias mitigation algorithms. That said, the clear advantage of Fair-SSL is that it requires only 10% of the labeled training data. To the best of our knowledge, this is the first SE work where semi-supervised techniques are used to fight against ethical bias in SE ML models.

Motivation & Objective

• To address the ethical bias in machine learning models prevalent in software engineering applications.
• To reduce dependency on large amounts of labeled data with trustworthy ground truth, which is often scarce and potentially biased.
• To explore whether semi-supervised learning techniques can effectively mitigate bias in classification models.
• To develop a framework that achieves fairness with minimal labeled data while maintaining model performance.
• To evaluate the effectiveness of semi-supervised learning in fairness mitigation, a novel application in software engineering.

Proposed method

• The framework uses four popular semi-supervised techniques as pseudo-labelers to generate predictions for unlabeled data.
• It takes only 10% of the training data as labeled and uses it to produce pseudo-labels for the remaining 90%.
• It applies data augmentation based on class and protected attribute balance, as proposed by Chakraborty et al. (FSE 2021), to synthetically generate new training samples.
• The final model is trained on the augmented, pseudo-labeled dataset and validated on test data.
• The approach is evaluated across ten datasets and three different classification learners.

Experimental results

Research questions

• RQ1 Can semi-supervised learning techniques effectively mitigate ethical bias in machine learning models?
• RQ2 How does the performance of Fair-SSL compare to state-of-the-art bias mitigation algorithms when using only 10% labeled data?
• RQ3 Does the use of pseudo-labeling and data augmentation improve fairness without requiring extensive labeled data?
• RQ4 Is Fair-SSL robust across diverse datasets and different classification learners?

Key findings

• Fair-SSL achieves fairness performance comparable to three state-of-the-art bias mitigation algorithms.
• The framework maintains strong model performance while using only 10% of the labeled training data.
• The use of pseudo-labeling and synthetic data augmentation effectively balances class and protected attribute distributions.
• Fair-SSL is the first software engineering work to apply semi-supervised learning for ethical bias mitigation in ML models.