[Paper Review] An Empirical Study of Modular Bias Mitigators and Ensembles

Michael Feffer, Martin Hirzel+4 more|arXiv (Cornell University)|Feb 1, 2022

Ethics and Social Impacts of AI4 citations

TL;DR

This paper proposes a modular framework combining 10 bias mitigators and 4 ensemble methods via an open-source library to improve fairness and stability in machine learning. Empirical evaluation across 13 datasets shows that ensembles—particularly stacking and boosting—enhance stability of fairness metrics, with optimal configurations depending on dataset size and initial fairness levels, leading to a guidance diagram for practitioners.

ABSTRACT

There are several bias mitigators that can reduce algorithmic bias in machine learning models but, unfortunately, the effect of mitigators on fairness is often not stable when measured across different data splits. A popular approach to train more stable models is ensemble learning. Ensembles, such as bagging, boosting, voting, or stacking, have been successful at making predictive performance more stable. One might therefore ask whether we can combine the advantages of bias mitigators and ensembles? To explore this question, we first need bias mitigators and ensembles to work together. We built an open-source library enabling the modular composition of 10 mitigators, 4 ensembles, and their corresponding hyperparameters. Based on this library, we empirically explored the space of combinations on 13 datasets, including datasets commonly used in fairness literature plus datasets newly curated by our library. Furthermore, we distilled the results into a guidance diagram for practitioners. We hope this paper will contribute towards improving stability in bias mitigation.

Motivation & Objective

• Address the instability of fairness metrics in bias mitigation, especially under data scarcity for underrepresented groups.
• Investigate whether ensemble methods can stabilize fairness metrics the same way they stabilize accuracy.
• Develop a modular, composable library enabling systematic combination of bias mitigators and ensembles for empirical study.
• Provide data-driven guidance for selecting optimal combinations of mitigators and ensembles based on dataset characteristics and fairness objectives.

Proposed method

• Built an open-source library (Lale) to enable modular composition of 10 AIF360-based bias mitigators and 4 scikit-learn-based ensemble methods with shared interfaces.
• Conducted empirical evaluation on 13 datasets, including standard fairness benchmarks and newly curated ones, measuring fairness using standardized outcomes and volatilities of disparate impact and statistical parity.
• Used standardized metrics (SDO, SDV, SSO, SSV) to compare fairness performance and stability across configurations, including ensemble-level vs. estimator-level mitigation.
• Applied a four-quadrant framework based on dataset size (>8,000 rows) and baseline unfairness (DI < 0.49) to group results and identify optimal configurations.
• Generated a decision guidance diagram (Fig. 6) by filtering top 33% results in both outcome and volatility metrics per dataset, then aggregating by quadrant and model configuration.

Experimental results

Research questions

• RQ1 Can modular ensembles improve the stability of fairness metrics in machine learning models?
• RQ2 How do different ensemble types (e.g., bagging, boosting, voting, stacking) interact with various bias mitigators to affect fairness outcomes?
• RQ3 What configurations of ensembles and mitigators yield the most stable and favorable fairness performance across diverse datasets?
• RQ4 How do dataset size and baseline fairness influence the optimal choice of ensemble and mitigator combination?
• RQ5 Can a data-driven guidance diagram help practitioners select effective combinations of ensembles and mitigators for new fairness-critical applications?

Key findings

• Ensembles, especially stacking and boosting, significantly reduce volatility in fairness metrics—e.g., SDV dropped to 0.229 for stacking with passthrough and final mitigation on one dataset.
• Stacking with base estimator mitigation and passthrough achieved the highest standardized disparate impact outcome (SDO = 0.832) and lowest volatility (SDV = 0.515) among configurations.
• For large and unfair datasets, boosting with in-estimator mitigators yielded the best fairness outcomes, with SDO of 0.803 and SDV of 0.642.
• Post-estimator mitigation with stacking was optimal for small, unfair datasets, achieving high SDO (0.670) and low SDV (0.506).
• The guidance diagram (Fig. 6) identifies context-dependent optimal configurations, showing that no single combination universally outperforms others across all datasets and metrics.
• Even with high fairness improvement, the best configurations depend heavily on dataset size and initial fairness, underscoring the need for context-aware selection.