[Paper Review] FairCanary: Rapid Continuous Explainable Fairness

Avijit Ghosh, Aalok Shanbhag+1 more|arXiv (Cornell University)|Jun 13, 2021

Ethics and Social Impacts of AI44 references4 citations

TL;DR

FairCanary introduces Quantile Demographic Drift (QDD), a threshold-free fairness metric that quantifies disparities in model prediction distributions across subgroups using quantile binning. By reusing existing instance-level explanations (e.g., SHAP or LIME), FairCanary enables rapid, continuous, and explainable fairness monitoring—achieving a 10x speedup over prior methods that recompute explanations from scratch.

ABSTRACT

Systems that offer continuous model monitoring have emerged in response to (1) well-documented failures of deployed Machine Learning (ML) and Artificial Intelligence (AI) models and (2) new regulatory requirements impacting these models. Existing monitoring systems continuously track the performance of deployed ML models and compute feature importance (a.k.a. explanations) for each prediction to help developers identify the root causes of emergent model performance problems. We present Quantile Demographic Drift (QDD), a novel model bias quantification metric that uses quantile binning to measure differences in the overall prediction distributions over subgroups. QDD is ideal for continuous monitoring scenarios, does not suffer from the statistical limitations of conventional threshold-based bias metrics, and does not require outcome labels (which may not be available at runtime). We incorporate QDD into a continuous model monitoring system, called FairCanary, that reuses existing explanations computed for each individual prediction to quickly compute explanations for the QDD bias metrics. This optimization makes FairCanary an order of magnitude faster than previous work that has tried to generate feature-level bias explanations.

Motivation & Objective

• To address the limitations of conventional fairness metrics that rely on arbitrary thresholds and outcome labels, which are often unavailable in production environments.
• To develop a continuous monitoring system that detects fairness drift in deployed ML models without requiring retraining or re-computation of explanations.
• To enable feature-level explanations for fairness metrics, providing actionable insights into the root causes of bias in real time.
• To support regulatory compliance with emerging AI governance frameworks that mandate post-market fairness monitoring.
• To improve the efficiency and scalability of fairness monitoring by reusing existing model explanation infrastructure.

Proposed method

• Introduces Quantile Demographic Drift (QDD), a fairness metric that compares prediction distributions across subgroups using quantile binning instead of fixed thresholds.
• Uses quantile-based binning to measure differences in cumulative prediction distributions, avoiding the statistical limitations of threshold-dependent metrics like demographic parity or equal opportunity.
• Reuses instance-level explanations (e.g., SHAP or LIME) computed for individual predictions to efficiently derive feature-level explanations for QDD, eliminating the need to recompute them.
• Integrates QDD into a continuous monitoring pipeline that tracks model inputs, outputs, and fairness metrics over time, with configurable alerting for drift.
• Employs a modular architecture that supports integration with existing ML monitoring stacks and allows for dynamic tuning of bin count and alert sensitivity.
• Validates the approach using synthetic case studies and real-world examples, demonstrating robustness to threshold selection and improved detection of fairness drift.

Experimental results

Research questions

• RQ1 Can a fairness metric be designed to be independent of arbitrary prediction thresholds while still detecting meaningful disparities in model behavior?
• RQ2 How can feature-level explanations for fairness metrics be computed efficiently in a continuous monitoring setting without recomputing explanations from scratch?
• RQ3 To what extent does QDD outperform conventional fairness metrics in detecting fairness drift when outcome labels are unavailable?
• RQ4 Can the reuse of existing instance-level explanations significantly reduce computational overhead in fairness monitoring?
• RQ5 How does QDD perform in detecting fairness drift across different types of data drift (e.g., gradual or sudden) in real-world deployment scenarios?

Key findings

• QDD successfully detects fairness drift in synthetic scenarios where conventional metrics like Statistical Parity Difference (SPD) and Disparate Impact (DI) fail to trigger alerts due to threshold sensitivity.
• In a two-day synthetic case study, SPD failed to detect bias on the second day across all tested thresholds, while DI only detected it at one threshold level, demonstrating the fragility of threshold-based metrics.
• FairCanary achieves a 10x speedup in computing feature-level explanations for fairness metrics compared to prior work that recomputes explanations from scratch.
• The reuse of existing instance-level explanations for QDD enables real-time, scalable fairness monitoring with minimal performance overhead.
• QDD is robust to the absence of outcome labels, making it suitable for production environments where labels may not be available at inference time.
• The system supports actionable insights through explainable bias detection, helping developers identify root causes of fairness issues in deployed models.