[Paper Review] An Effective Meaningful Way to Evaluate Survival Models

Shi-ang Qi, Neeraj Kumar+6 more|arXiv (Cornell University)|Jun 1, 2023

Statistical Methods and Inference4 citations

TL;DR

This paper proposes MAE-PO, a novel metric for evaluating survival models using pseudo-observations to handle censored data, which outperforms existing MAE variants in both ranking model performance and proximity to the true Mean Absolute Error. The method estimates pseudo-observations based on Kaplan-Meier survival probabilities and applies a weighted confidence scheme, demonstrating superior accuracy on semi-synthetic datasets with diverse censoring mechanisms.

ABSTRACT

One straightforward metric to evaluate a survival prediction model is based on the Mean Absolute Error (MAE) -- the average of the absolute difference between the time predicted by the model and the true event time, over all subjects. Unfortunately, this is challenging because, in practice, the test set includes (right) censored individuals, meaning we do not know when a censored individual actually experienced the event. In this paper, we explore various metrics to estimate MAE for survival datasets that include (many) censored individuals. Moreover, we introduce a novel and effective approach for generating realistic semi-synthetic survival datasets to facilitate the evaluation of metrics. Our findings, based on the analysis of the semi-synthetic datasets, reveal that our proposed metric (MAE using pseudo-observations) is able to rank models accurately based on their performance, and often closely matches the true MAE -- in particular, is better than several alternative methods.

Motivation & Objective

• To address the challenge of evaluating survival models when test sets include right-censored individuals, where true event times are unknown.
• To develop a meaningful and accurate metric for Mean Absolute Error (MAE) that accounts for censored observations in survival prediction.
• To generate realistic semi-synthetic survival datasets with known ground-truth event times to enable unbiased evaluation of evaluation metrics.
• To compare multiple MAE-inspired metrics and identify the most reliable estimator for model ranking and error estimation.
• To provide a reproducible codebase for MAE-PO and related metrics to support future research.

Proposed method

• Proposes MAE-PO, a metric that uses pseudo-observations to estimate the contribution of censored subjects to the group-level Kaplan-Meier estimator.
• Applies a weighted scheme to represent the confidence in each pseudo-observation, improving robustness to censoring.
• Employs the pseudo-observation de-censoring technique (Andersen et al., 2003) to transform censored survival data into usable point estimates for error computation.
• Generates semi-synthetic survival datasets by simulating realistic feature and censoring distributions, preserving the structure of real-world data.
• Uses the ground-truth event times in these synthetic datasets to compute the true MAE, enabling benchmarking of evaluation metrics.
• Compares six MAE variants (including MAE-uncensored, MAE-margin, MAE-PO, IPCW-D, IPCW-T, and MAE-Pop-PO) across multiple datasets and censoring mechanisms.

Experimental results

Research questions

• RQ1 How do different MAE-inspired metrics perform in ranking survival models when censored data are present?
• RQ2 Which metric most closely approximates the true MAE in the presence of censored observations?
• RQ3 Can a pseudo-observation-based approach (MAE-PO) provide a more accurate and reliable evaluation than existing alternatives?
• RQ4 How does the performance of evaluation metrics vary across different censoring distributions (e.g., exponential, feature-independent, feature-dependent)?
• RQ5 To what extent does the proposed semi-synthetic dataset generation method enable valid and reproducible evaluation of survival model metrics?

Key findings

• MAE-PO consistently ranks models most accurately across all datasets and censoring mechanisms, correctly identifying the top-three models in all but one case.
• MAE-PO achieves the closest proximity to the true MAE, with significantly smaller error than alternatives on METABRIC and MIMIC-A datasets (p < 0.05).
• MAE-uncensored performs poorly under high censoring, while MAE-PO maintains strong performance even in high-censoring scenarios.
• MAE-margin performs well in some settings but fails to identify top models in MIMIC-H, whereas MAE-PO correctly identifies GBM-C as the top-performing model.
• MAE-IPCW-D shows high variance in proximity to true MAE due to sensitivity to later uncensored subjects, limiting its reliability.
• MAE-Pop-PO shows no advantage in ranking or error approximation, performing worse than both MAE-margin and MAE-PO.