[Paper Review] Limitations of ROC on Imbalanced Data: Evaluation of LVAD Mortality Risk Scores

Faezeh Movahedi, Rema Padman+1 more|arXiv (Cornell University)|Oct 29, 2020

Imbalanced Data Classification Techniques22 references4 citations

TL;DR

This paper demonstrates that ROC curves can give a misleadingly optimistic view of classifier performance in imbalanced medical data, such as 90-day LVAD mortality prediction, where only 8% of patients die. It proposes the Precision-Recall Curve (PRC) as a superior, complementary metric that better evaluates performance on the minority (death) class, showing significantly lower AUCs for both HMRS and Random Forest classifiers on PRC compared to ROC, revealing their poor real-world predictive power for mortality.

ABSTRACT

Objective: This study illustrates the ambiguity of ROC in evaluating two classifiers of 90-day LVAD mortality. This paper also introduces the precision recall curve (PRC) as a supplemental metric that is more representative of LVAD classifiers performance in predicting the minority class. Background: In the LVAD domain, the receiver operating characteristic (ROC) is a commonly applied metric of performance of classifiers. However, ROC can provide a distorted view of classifiers ability to predict short-term mortality due to the overwhelmingly greater proportion of patients who survive, i.e. imbalanced data. Methods: This study compared the ROC and PRC for the outcome of two classifiers for 90-day LVAD mortality for 800 patients (test group) recorded in INTERMACS who received a continuous-flow LVAD between 2006 and 2016 (mean age of 59 years; 146 females vs. 654 males) in which mortality rate is only %8 at 90-day (imbalanced data). The two classifiers were HeartMate Risk Score (HMRS) and a Random Forest (RF). Results: The ROC indicates fairly good performance of RF and HRMS classifiers with Area Under Curves (AUC) of 0.77 vs. 0.63, respectively. This is in contrast with their PRC with AUC of 0.43 vs. 0.16 for RF and HRMS, respectively. The PRC for HRMS showed the precision rapidly dropped to only 10% with slightly increasing sensitivity. Conclusion: The ROC can portray an overly-optimistic performance of a classifier or risk score when applied to imbalanced data. The PRC provides better insight about the performance of a classifier by focusing on the minority class.

Motivation & Objective

• To highlight the limitations of ROC in evaluating classifiers on imbalanced medical data, particularly in LVAD mortality prediction.
• To demonstrate that ROC can provide a deceptively optimistic assessment of classifier performance when the minority class (death) is rare.
• To introduce the Precision-Recall Curve (PRC) as a more appropriate and informative evaluation metric for minority class prediction in imbalanced datasets.
• To compare the performance of two LVAD mortality classifiers—HeartMate Risk Score (HMRS) and Random Forest (RF)—using both ROC and PRC to reveal discrepancies in performance assessment.
• To advocate for the adoption of PRC as a standard supplementary metric in clinical risk model evaluation, especially in low-prevalence outcomes.

Proposed method

• Evaluated two classifiers—HMRS and Random Forest—on a test set of 800 LVAD patients from the INTERMACS registry with 90-day mortality as the outcome.
• Computed ROC AUC and PRC AUC for both classifiers to compare their performance on imbalanced data with 92% survival and 8% mortality.
• Used a standard threshold-based classification approach to convert predicted probabilities into hard labels for performance evaluation.
• Employed a 70/30 train-test split on 11,967 patients from INTERMACS to train the Random Forest model using 235 pre-LVAD clinical variables.
• Analyzed classifier performance by focusing on precision and recall, particularly emphasizing the minority class (DEAD) to highlight PRC's sensitivity to imbalance.
• Contrasted the interpretation of ROC (balanced focus on both classes) with PRC (focused on minority class) to illustrate why PRC better reflects real-world clinical utility.

Experimental results

Research questions

• RQ1 How does the performance of ROC compare to PRC in evaluating LVAD mortality risk classifiers on imbalanced data?
• RQ2 To what extent does ROC overestimate the predictive performance of classifiers when the outcome class is highly imbalanced, such as 90-day mortality post-LVAD?
• RQ3 Can the Precision-Recall Curve provide a more accurate and clinically meaningful assessment of classifier performance for predicting rare adverse outcomes like LVAD mortality?
• RQ4 Why do traditional metrics like AUC-ROC fail to reflect the true clinical utility of risk scores in low-prevalence outcomes?
• RQ5 What are the implications of using ROC versus PRC for clinical decision-making in pre-LVAD risk assessment?

Key findings

• The ROC AUC for the Random Forest classifier was 0.77, suggesting moderate to good performance, while its PRC AUC was only 0.43, indicating poor performance on the minority class.
• The HMRS had a ROC AUC of 0.63, suggesting fair performance, but its PRC AUC was just 0.16, revealing very weak predictive ability for 90-day mortality.
• The PRC revealed that HMRS precision dropped rapidly to 10% even with modest sensitivity, indicating high false positive rates and poor reliability in identifying true deaths.
• The discrepancy between ROC and PRC results shows that ROC can give a dangerously misleadingly optimistic view of classifier performance in imbalanced data settings.
• The study concludes that PRC is a more informative and realistic metric for evaluating classifiers in clinical settings with rare outcomes, such as post-LVAD mortality.
• The findings suggest that risk scores like HMRS, which are widely used, may be overestimated in performance due to reliance on ROC, and should be re-evaluated using PRC for better clinical decision support.