[Paper Review] cutpointr: Improved Estimation and Validation of Optimal Cutpoints in R

Christian Thiele, Gerrit Hirschfeld|arXiv (Cornell University)|Feb 21, 2020

Statistical Methods and Inference16 references36 citations

TL;DR

The cutpointr package provides robust methods to estimate optimal binary cutpoints and assess their out-of-sample performance in R, using bootstrapping and various smoothing techniques.

ABSTRACT

'Optimal cutpoints' for binary classification tasks are often established by testing which cutpoint yields the best discrimination, for example the Youden index, in a specific sample. This results in 'optimal' cutpoints that are highly variable and systematically overestimate the out-of-sample performance. To address these concerns, the cutpointr package offers robust methods for estimating optimal cutpoints and the out-of-sample performance. The robust methods include bootstrapping and smoothing based on kernel estimation, generalized additive models, smoothing splines, and local regression. These methods can be applied to a wide range of binary-classification and cost-based metrics. cutpointr also provides mechanisms to utilize user-defined metrics and estimation methods. The package has capabilities for parallelization of the bootstrapping, including reproducible random number generation. Furthermore, it is pipe-friendly, for example for compatibility with functions from tidyverse. Various functions for plotting receiver operating characteristic curves, precision recall graphs, bootstrap results and other representations of the data are included. The package contains example data from a study on psychological characteristics and suicide attempts suitable for applying binary classification algorithms.

Motivation & Objective

• Motivate robust estimation of optimal cutpoints beyond in-sample optimization.
• Provide methods to assess out-of-sample performance via bootstrap validation.
• Offer scalable, tidy, pipe-friendly tools for integration with workflows like tidyverse.
• Support a wide range of binary-classification metrics and user-defined metrics/methods.

Proposed method

• Define metrics for discriminatory ability (e.g., Youden index, Se/Sp, PPV/NPV, and others).
• Provide multiple cutpoint estimation methods: nonparametric empirical, LOESS smoothing, spline smoothing, GAM smoothing, bootstrap, kernel methods, and parametric normal method.
• Allow user-defined metrics and estimation methods; return results in tidy data frames.
• Enable parallelized bootstrapping with reproducible random numbers; integrate with tidyverse pipelines.
• Return ROC curves and options to plot bootstrap results, ROC/PR curves, and other representations.

Experimental results

Research questions

• RQ1 How can optimal cutpoints be estimated robustly across different distributions and sample sizes?
• RQ2 Does bootstrap validation improve the reliability of out-of-sample performance estimates for cutpoints?
• RQ3 Which estimation methods (empirical, smoothing, GAM, bootstrap, kernel, normal) perform best under various data-generating scenarios?
• RQ4 How can cutpoint estimation be made pipeline-friendly and scalable for large datasets?
• RQ5 Can users flexibly define metrics and still obtain coherent in-sample and out-of-sample summaries?

Key findings

• Advanced methods (bootstrap and GAM smoothing) generally outperform simple empirical approaches, especially for non-normal data.
• The Normal parametric method performs well only under normality and can yield erroneous cutpoints for non-normal distributions.
• Empirical and LOESS methods tend to have higher error under normal data but may perform differently under other conditions; overall, more robust methods are preferable.
• Bootstrapping provides out-of-sample performance estimates and variability without requiring tuning parameters, though it can be computationally intensive without parallelization.
• Cutpointr emphasizes parallelizable, tidy outputs and integrates smoothly with tidyverse workflows for practical use.