[Paper Review] Justification-Based Reliability in Machine Learning

Nurali Virani, Naresh Iyer+1 more|arXiv (Cornell University)|Nov 18, 2019

Adversarial Robustness in Machine Learning17 references4 citations

TL;DR

This paper introduces a justification-based reliability framework for machine learning models by extending Plato's Justified True Belief (JTB) theory to quantify epistemic uncertainty in individual predictions. It proposes neighborhood operators (e.g., k-NN, ℓp-norm, ε-NN) to gather training data support for predictions, enabling classification into 'I know' (IK), 'I may know' (IMK), or 'I don’t know' (IDK) regions, with experiments showing superior robustness to adversarial perturbations and improved reliability over baseline thresholding methods.

ABSTRACT

With the advent of Deep Learning, the field of machine learning (ML) has surpassed human-level performance on diverse classification tasks. At the same time, there is a stark need to characterize and quantify reliability of a model's prediction on individual samples. This is especially true in application of such models in safety-critical domains of industrial control and healthcare. To address this need, we link the question of reliability of a model's individual prediction to the epistemic uncertainty of the model's prediction. More specifically, we extend the theory of Justified True Belief (JTB) in epistemology, created to study the validity and limits of human-acquired knowledge, towards characterizing the validity and limits of knowledge in supervised classifiers. We present an analysis of neural network classifiers linking the reliability of its prediction on an input to characteristics of the support gathered from the input and latent spaces of the network. We hypothesize that the JTB analysis exposes the epistemic uncertainty (or ignorance) of a model with respect to its inference, thereby allowing for the inference to be only as strong as the justification permits. We explore various forms of support (for e.g., k-nearest neighbors (k-NN) and l_p-norm based) generated for an input, using the training data to construct a justification for the prediction with that input. Through experiments conducted on simulated and real datasets, we demonstrate that our approach can provide reliability for individual predictions and characterize regions where such reliability cannot be ascertained.

Motivation & Objective

• To formalize the concept of justified belief from epistemology for machine learning models to improve individual prediction reliability.
• To distinguish epistemic uncertainty (lack of knowledge due to missing information) from aleatoric uncertainty (inherent data variability) in model predictions.
• To develop domain-agnostic neighborhood operators that gather evidence from training data in input and latent spaces to justify predictions.
• To characterize input regions as 'I know' (IK), 'I may know' (IMK), or 'I don’t know' (IDK) based on support quality and data density.
• To evaluate the method’s robustness under adversarial perturbations and noise, using new metrics like F_IK and A_IK for reliability assessment.

Proposed method

• Extends the Justified True Belief (JTB) framework from epistemology to define model predictions as beliefs that require justification to count as knowledge.
• Uses k-NN, ℓp-norm, and ε-NN operators to collect support from training data points in input and hidden layer spaces for a given input.
• Classifies predictions into three reliability zones: IK (trustworthy), IMK (ambiguous due to overlapping support), and IDK (anomalous, no nearby support).
• Applies neighborhood operators across multiple layers (e.g., logits and intermediate layers) to improve robustness and justification quality.
• Employs new metrics—F_IK (fraction of IK predictions that are correct) and A_IK (accuracy of IK predictions)—to evaluate reliability under perturbations.
• Uses Platt scaling and neighborhood-based trust scores to calibrate predictions, avoiding reliance on softmax confidence thresholds.

Experimental results

Research questions

• RQ1 Can the JTB framework be adapted to quantify epistemic uncertainty in machine learning models for individual predictions?
• RQ2 How do different neighborhood operators (k-NN, ℓp-norm, ε-NN) affect the reliability and robustness of model predictions?
• RQ3 Can the method distinguish between regions of extrapolation (IDK), confusion (IMK), and trust (IK) in input space?
• RQ4 How does the proposed method compare to baseline thresholding of softmax scores in terms of robustness to adversarial and benign perturbations?
• RQ5 To what extent does gathering support from multiple network layers improve reliability and reduce overconfidence in predictions?

Key findings

• The ε-NN and H-2 neighborhood operators achieved high F_IK and A_IK under large adversarial perturbations, outperforming k-NN and baseline methods.
• Baseline models had high F_IK but poor A_IK on IK predictions under perturbations, indicating overconfidence, whereas the proposed method correctly identified IDK regions.
• Support gathered from at least two layers (e.g., logits and intermediate layers) improved robustness compared to using only the logits layer.
• The method successfully identified IDK regions with empty neighborhoods, indicating anomalous or novel inputs, and IMK regions with impure support, signaling need for additional information.
• The approach demonstrated superior reliability on simulated data, time-series, and image data, with improved performance in adversarial robustness and anomaly detection.
• The H-2 and ε-NN operators showed lower coverage than k-NN but significantly better reliability under large perturbations, highlighting trade-offs between coverage and robustness.