[Paper Review] Can You Trust This Prediction? Auditing Pointwise Reliability After Learning

Peter Schulam, Suchi Saria|arXiv (Cornell University)|Jan 2, 2019

Adversarial Robustness in Machine Learning51 references37 citations

TL;DR

The paper introduces Resampling Uncertainty Estimation (RUE), an auditing method to quantify per-prediction reliability after training by approximating bootstrap-like resampling using gradients and Hessian information. RUE detects inaccurate predictions more effectively than several post-hoc methods and yields competitive predictive distributions without requiring changes to the training procedure.

ABSTRACT

To use machine learning in high stakes applications (e.g. medicine), we need tools for building confidence in the system and evaluating whether it is reliable. Methods to improve model reliability often require new learning algorithms (e.g. using Bayesian inference to obtain uncertainty estimates). An alternative is to audit a model after it is trained. In this paper, we describe resampling uncertainty estimation (RUE), an algorithm to audit the pointwise reliability of predictions. Intuitively, RUE estimates the amount that a prediction would change if the model had been fit on different training data. The algorithm uses the gradient and Hessian of the model's loss function to create an ensemble of predictions. Experimentally, we show that RUE more effectively detects inaccurate predictions than existing tools for auditing reliability subsequent to training. We also show that RUE can create predictive distributions that are competitive with state-of-the-art methods like Monte Carlo dropout, probabilistic backpropagation, and deep ensembles, but does not depend on specific algorithms at train-time like these methods do.

Motivation & Objective

• Motivate the need for reliability tools in high-stakes ML applications beyond traditional held-out error metrics.
• Introduce RUE as a post-learning auditing method that estimates per-prediction uncertainty without retraining.
• Show that RUE aligns with density and local-fit reliability criteria using a model-dependent similarity measure.
• Demonstrate RUE's effectiveness at detecting mispredictions and producing competitive predictive distributions.

Proposed method

• Compute an uncertainty score for each test input by sampling predictions from an ensemble of parameters generated via a bootstrap-like procedure.
• Use the gradient and Hessian of the training loss to form an approximate map from bootstrap weights to updated parameters (A = H̃^{-1}L).
• Draw bootstrap-like weight vectors w from a multinomial distribution and create parameter samples θ* = θ̂ − A(w − w0).
• Predict on test inputs with each θ* to form an ensemble and compute the column-wise prediction variance as the uncertainty score σ̂_RUE^2(x).
• Dampen the Hessian with λI to ensure invertibility and discuss connections to bootstrap, Laplace approximation, and robust statistics.
• Relate σ̂_RUE^2(x) to a model-dependent kernel k_RUE that encodes input similarity via gradients and curvature.

Experimental results

Research questions

• RQ1 Can post-training auditing reveal pointwise unreliability in predictions without altering the training process?
• RQ2 How does an RUE-based uncertainty score compare to existing post-hoc methods (Laplace, KDE, Bootstrap SGD) in detecting incorrect predictions?
• RQ3 Can RUE produce predictive distributions competitive with integrated uncertainty methods that are trained with uncertainty objectives?
• RQ4 What are the theoretical connections between RUE, bootstrap, and robust statistics, and how do they manifest empirically?

Key findings

• RUE more effectively detects inaccurate predictions than Laplace, KDE, and Bootstrap SGD across most datasets and error thresholds.
• RUE can generate predictive distributions competitive with state-of-the-art integrated methods, despite not altering the training procedure.
• RUE’s uncertainty score implements density and local-fit criteria via a model-dependent inner product, improving interpretability over Euclidean-distance based approaches.
• RUE provides a practical auditing tool that pairs with standard training pipelines, avoiding train-time algorithm dependencies required by Bayesian or ensemble methods.
• The paper discusses scalability considerations and practical strategies (e.g., Hessian-vector products, dampening, and possible approximations) to enable deployment.