[Paper Review] (Un)fairness in Post-operative Complication Prediction Models

Sandhya Tripathi, Bradley A. Fritz+4 more|arXiv (Cornell University)|Nov 3, 2020

Artificial Intelligence in Healthcare and Education43 references4 citations

TL;DR

This paper investigates fairness in post-operative complication prediction models, proposing a model-card-like framework with propensity score matching and decision-tree-guided utility analysis to detect and mitigate bias. It finds significant disparities in algorithmic utility across sex and race, particularly in AKI prediction, highlighting the need for targeted data collection and model transparency to prevent exacerbating health disparities.

ABSTRACT

With the current ongoing debate about fairness, explainability and transparency of machine learning models, their application in high-impact clinical decision-making systems must be scrutinized. We consider a real-life example of risk estimation before surgery and investigate the potential for bias or unfairness of a variety of algorithms. Our approach creates transparent documentation of potential bias so that the users can apply the model carefully. We augment a model-card like analysis using propensity scores with a decision-tree based guide for clinicians that would identify predictable shortcomings of the model. In addition to functioning as a guide for users, we propose that it can guide the algorithm development and informatics team to focus on data sources and structures that can address these shortcomings.

Motivation & Objective

• To assess fairness and potential bias in machine learning models predicting post-operative complications.
• To develop a transparent, auditable framework for detecting disparate mistreatment across protected characteristics like sex and race.
• To guide clinicians and developers in identifying data gaps and model shortcomings through utility-based decision trees.
• To prevent unintended health disparities by diagnosing algorithmic unfairness before clinical deployment.
• To promote equitable AI in healthcare by linking model performance to real-world clinical utility across subgroups.

Proposed method

• The authors use a model-card-like documentation framework to transparently record potential biases in clinical prediction models.
• They apply propensity score matching to detect and control for surrogate variables that may encode sensitive attributes.
• A decision-tree-based utility analysis identifies patients who gain most or least from model predictions, stratified by sex, race, and surgery type.
• The method compares classifier performance across subgroups using in-sample and test-sample predictions to detect disparities in accuracy and utility.
• They validate findings with domain experts to identify key clinical covariates and assess whether performance differences stem from real clinical variation or algorithmic bias.
• The approach uses logistic regression and random forest classifiers, with minimal cost-complexity pruning to ensure model interpretability.

Experimental results

Research questions

• RQ1 How do post-operative complication prediction models perform differently across sex and racial groups?
• RQ2 To what extent do sensitive attributes like sex and race influence model utility, even when not explicitly included?
• RQ3 Can propensity score matching detect hidden biases from surrogate variables in clinical prediction models?
• RQ4 What are the key clinical and demographic factors driving differential algorithmic utility?
• RQ5 How can clinicians and developers use utility-based decision trees to improve fairness and transparency in AI-driven clinical decision support?

Key findings

• Significant disparities in model performance were found across sexes in acute kidney injury (AKI) prediction, with males consistently receiving lower utility from the model.
• Racial disparities were smaller but still present, with Black patients showing higher utility gains from the model compared to White patients.
• The decision tree analysis revealed that race and age were major drivers of algorithmic utility, even when not explicitly included as features.
• Propensity score matching identified potential surrogate variables that may encode sensitive characteristics, suggesting hidden bias in model inputs.
• Despite including sex and race in the base model, the algorithm did not select them as top predictors, indicating that their influence may be mediated through other covariates.
• The study demonstrates that even with careful model design, residual unfairness can persist due to unmeasured or unbalanced data, especially in underrepresented groups.