[Paper Review] Calibrated neighborhood aware confidence measure for deep metric learning

Maryna Karpusha, Sunghee Yun+1 more|arXiv (Cornell University)|Jun 8, 2020

Domain Adaptation and Few-Shot Learning46 references4 citations

TL;DR

This paper proposes NED (Normalized Exponential of Distances), a calibrated and interpretable confidence measure for deep metric learning that approximates class-wise data distributions using Gaussian kernel smoothing in the embedding space. It improves robustness and generalization across clean, corrupted, and adversarial test data with minimal hyperparameter tuning, consistently outperforming baselines on four benchmark datasets.

ABSTRACT

Deep metric learning has gained promising improvement in recent years following the success of deep learning. It has been successfully applied to problems in few-shot learning, image retrieval, and open-set classifications. However, measuring the confidence of a deep metric learning model and identifying unreliable predictions is still an open challenge. This paper focuses on defining a calibrated and interpretable confidence metric that closely reflects its classification accuracy. While performing similarity comparison directly in the latent space using the learned distance metric, our approach approximates the distribution of data points for each class using a Gaussian kernel smoothing function. The post-processing calibration algorithm with proposed confidence metric on the held-out validation dataset improves generalization and robustness of state-of-the-art deep metric learning models while provides an interpretable estimation of the confidence. Extensive tests on four popular benchmark datasets (Caltech-UCSD Birds, Stanford Online Product, Stanford Car-196, and In-shop Clothes Retrieval) show consistent improvements even at the presence of distribution shifts in test data related to additional noise or adversarial examples.

Motivation & Objective

• To address the lack of reliable, interpretable confidence estimation in deep metric learning models.
• To improve model robustness under distribution shifts, such as image corruptions and adversarial attacks.
• To provide a post-processing calibration method that generalizes well without retraining the backbone network.
• To enable accurate confidence estimation that correlates strongly with prediction accuracy.
• To reduce dependency on hyperparameter sensitivity common in k-NN based confidence scores.

Proposed method

• NED computes confidence as the normalized sum of exponentials of distances to the k nearest neighbors in the embedding space.
• The method uses a Gaussian kernel smoothing approximation to model the conditional probability of a sample belonging to a given class.
• A single temperature parameter T is calibrated on a held-out validation set to improve probability estimates, analogous to temperature scaling in classification.
• The confidence score is derived from Bayes' theorem, linking the distance-based neighborhood information to the true correctness likelihood.
• The approach is applied post-hoc to pre-trained deep metric learning models, requiring no architecture changes or retraining.
• The method is theoretically grounded in kernel density estimation and statistical learning theory, ensuring generalization guarantees.

Experimental results

Research questions

• RQ1 Can a post-hoc confidence measure be developed for deep metric learning that correlates strongly with prediction accuracy?
• RQ2 How can confidence estimation be made robust to distribution shifts such as image corruptions and adversarial examples?
• RQ3 Does a neighborhood-aware confidence measure outperform standard k-NN and weighted k-NN baselines in terms of calibration and accuracy?
• RQ4 To what extent does the proposed NED method reduce sensitivity to hyperparameter choices like k?
• RQ5 Can a single, calibrated temperature parameter improve confidence calibration across diverse test distributions?

Key findings

• On the CUB-200 dataset, NED achieved 64.8% average accuracy on corrupted test images, outperforming 1NN by 12% and kNN by 4.3%.
• On the CARS196 dataset, NED achieved 77.2% accuracy with only 2.3% ECE on corrupted data, significantly outperforming kNN (76.0% accuracy, 14.1% ECE).
• On adversarial examples using PGD with ε=0.3, NED maintained 78.0% accuracy and 2.9% ECE, outperforming 1NN (74.4% accuracy, 10.9% ECE).
• NED showed consistent improvements across all four benchmark datasets: CUB-200, CARS196, SOP, and In-shop, with lower ECE and higher accuracy than all baselines.
• The method demonstrated robustness to k selection, with minimal accuracy drop beyond k=10, indicating low sensitivity to hyperparameter tuning.
• The calibrated temperature parameter T was found to improve probability estimates by better approximating the true data distribution, validated through theoretical and empirical analysis.