[Paper Review] To Trust Or Not To Trust A Classifier

Heinrich Jiang, Been Kim+2 more|arXiv (Cornell University)|May 30, 2018

Topological and Geometric Data AnalysisComputer Science33 references185 citations

TL;DR

The paper introduces the trust score, a data-distribution-based metric that compares a classifier’s prediction to a modified nearest-neighbor classifier after density-based filtering, showing it better identifies trustworthy and suspicious predictions than the model’s confidence. It provides non-asymptotic guarantees and empirical evidence across datasets and representations.

ABSTRACT

Knowing when a classifier's prediction can be trusted is useful in many applications and critical for safely using AI. While the bulk of the effort in machine learning research has been towards improving classifier performance, understanding when a classifier's predictions should and should not be trusted has received far less attention. The standard approach is to use the classifier's discriminant or confidence score; however, we show there exists an alternative that is more effective in many situations. We propose a new score, called the trust score, which measures the agreement between the classifier and a modified nearest-neighbor classifier on the testing example. We show empirically that high (low) trust scores produce surprisingly high precision at identifying correctly (incorrectly) classified examples, consistently outperforming the classifier's confidence score as well as many other baselines. Further, under some mild distributional assumptions, we show that if the trust score for an example is high (low), the classifier will likely agree (disagree) with the Bayes-optimal classifier. Our guarantees consist of non-asymptotic rates of statistical consistency under various nonparametric settings and build on recent developments in topological data analysis.

Motivation & Objective

• Motivate safe and reliable use of ML by assessing when classifier predictions can be trusted.
• Propose a trust score that compares a classifier to a modified nearest-neighbor reference built from high-density regions.
• Provide non-asymptotic guarantees under various nonparametric settings, including manifold and near-manifold data.
• Empirically demonstrate the trust score's effectiveness across multiple datasets, models, and representations.

Proposed method

• Define the alpha-high-density-set per class by filtering the lowest-density samples using k-NN density estimates.
• Compute the trust score as the ratio of distance from the test point to the nearest alpha-high-density-set of a different class to the distance to the alpha-high-density-set of the predicted class.
• Provide algorithms for estimating the alpha-high-density-set (Algorithm 1) and for computing the trust score (Algorithm 2).
• Allow distances to be computed in any representation (raw input, embeddings, or intermediate network layers).
• Use two hyperparameters: k (neighbors) and alpha (density fraction), with cross-validation used to select alpha in practice.
• Prove non-asymptotic consistency results for Algorithm 1 in full-dimensional and manifold settings, and extend to full-dimensional noise cases; prove trust-score guarantees (Theorem 4) under well-behaved class margins.

Experimental results

Research questions

• RQ1 Can a trust score based on density-filtered nearest-neighbor distances better indicate when a classifier’s prediction should be trusted compared to the classifier’s own confidence?
• RQ2 Under what conditions do high/low trust scores correspond to agreement/disagreement with the Bayes-optimal classifier, and what are the estimation rates in various data geometries (full-dimensional, manifold, noisy manifold)?
• RQ3 How does the trust score perform across different data representations and model families (neural networks, random forests, logistic regression) and dimensionalities?
• RQ4 What are the theoretical guarantees and practical implications of using density-based trust scores for reliability and safety in ML predictions?

Key findings

• The trust score often yields higher precision in identifying correctly classified examples than the model’s confidence, especially in low- to mid-dimensional spaces.
• Theoretical guarantees show high trust scores tend to agree with the Bayes-optimal classifier, while low scores tend to disagree under mild distributional assumptions.
• Estimation rates for the alpha-high-density-set depend on intrinsic dimension; on manifolds, rates scale with the manifold dimension d and can be independent of ambient dimension D, even with near-manifold noise.
• Algorithm 1 achieves Hausdorff-consistent estimation of the alpha-high-density-set under regularity assumptions, with rates that adapt to manifold structure.
• Algorithm 2 provides probabilistic guarantees that the trust score correctly indicates alignment with the Bayes-optimal classifier, given margin conditions.
• Empirically, the trust score outperforms model confidence across UCI datasets and CPU/GPU benchmarks, and across representations including intermediate neural network layers.