[Paper Review] The Brier Score under Administrative Censoring: Problems and Solutions

Håvard Kvamme, Ørnulf Borgan|arXiv (Cornell University)|Dec 18, 2019

Statistical Methods and Inference19 references4 citations

TL;DR

This paper identifies critical flaws in the inverse probability of censoring weighted (IPCW) Brier score under administrative censoring—where censoring times are known—showing it can be biased and misleading when censoring times are predictable from covariates. To resolve this, the authors propose the administrative Brier score, a novel evaluation metric that avoids censoring distribution estimation, remains valid under administrative censoring, and provides reliable performance assessment for survival models, especially in machine learning applications like customer churn prediction.

ABSTRACT

The Brier score is commonly used for evaluating probability predictions. In survival analysis, with right-censored observations of the event times, this score can be weighted by the inverse probability of censoring (IPCW) to retain its original interpretation. It is common practice to estimate the censoring distribution with the Kaplan-Meier estimator, even though it assumes that the censoring distribution is independent of the covariates. This paper discusses the general impact of the censoring estimates on the Brier score and shows that the estimation of the censoring distribution can be problematic. In particular, when the censoring times can be identified from the covariates, the IPCW score is no longer valid. For administratively censored data, where the potential censoring times are known for all individuals, we propose an alternative version of the Brier score. This administrative Brier score does not require estimation of the censoring distribution and is valid even if the censoring times can be identified from the covariates.

Motivation & Objective

• To identify and diagnose the bias in the IPCW Brier score when censoring times are predictable from covariates in administrative censoring settings.
• To demonstrate that binary classifiers, commonly used in machine learning, can achieve deceptively good IPCW Brier scores due to this bias, even when their predictions are inferior to survival models.
• To develop a new evaluation metric that does not rely on censoring distribution estimation and remains valid under administrative censoring.
• To provide a robust, practical alternative to IPCW-based metrics for evaluating time-to-event predictions in survival analysis, especially for neural network and machine learning methods.

Proposed method

• Propose the administrative Brier score as a non-IPCW alternative that directly uses known censoring times $C^*_i$ without estimating the censoring distribution.
• Define the administrative Brier score as the mean squared error between predicted survival probabilities and observed event indicators, adjusted only by the known administrative censoring times.
• Show that the minimizer of the administrative Brier score corresponds to the true survival function $S_i(t)$ when $C^*_i > t$, and zero otherwise, ensuring validity under covariate-dependent censoring.
• Demonstrate that binary classifiers (BCE method) trained on censored data minimize a loss function whose minimizer is equivalent to the IPCW Brier score under administrative censoring, explaining the bias.
• Use simulations and a real-world KKBox churn dataset to compare the IPCW Brier score and the proposed administrative Brier score under varying censoring predictability.
• Establish theoretical and empirical equivalence between the administrative Brier score and the minimizer of a non-IPCW loss function, proving its validity and robustness.

Experimental results

Research questions

• RQ1 Does the IPCW Brier score produce biased evaluations when censoring times are predictable from covariates in administrative censoring settings?
• RQ2 Why do binary classifiers often outperform survival models in IPCW Brier score evaluations despite worse predictive accuracy?
• RQ3 Can a Brier score variant be constructed that avoids censoring distribution estimation and remains valid under administrative censoring?
• RQ4 How does the performance of the administrative Brier score compare to the IPCW Brier score in simulations and real data with known censoring times?

Key findings

• The IPCW Brier score becomes biased when censoring times $C^*_i$ are predictable from covariates, leading to misleadingly favorable scores for binary classifiers.
• Binary classifiers trained on censored data minimize a loss function whose minimizer is equivalent to the IPCW Brier score under administrative censoring, explaining their inflated performance on this metric.
• The proposed administrative Brier score does not require estimation of the censoring distribution and remains valid even when censoring times are fully predictable from covariates.
• In simulations and real data (KKBox churn dataset), the administrative Brier score correctly identifies superior survival models, while the IPCW Brier score fails to do so under covariate-dependent censoring.
• The administrative Brier score is theoretically grounded as the minimizer of a non-IPCW loss function, ensuring it estimates the true survival probability without bias.
• The administrative Brier score is simpler to use and more reliable than IPCW-based metrics in machine learning applications involving administrative censoring.