[Paper Review] Uncertainty Estimates for Ordinal Embeddings

Michael Lohaus, Philipp Hennig+1 more|arXiv (Cornell University)|Jun 27, 2019

Gaussian Processes and Bayesian Inference27 references4 citations

TL;DR

This paper proposes bootstrap and Bayesian uncertainty estimation methods for ordinal embeddings under limited, noisy triplet data. By generating distributions over embeddings, the approach enables well-calibrated uncertainty quantification for triplet predictions, embedding parameters, and psychophysics applications—though it shows limited benefit for active learning in tested scenarios.

ABSTRACT

To investigate objects without a describable notion of distance, one can gather ordinal information by asking triplet comparisons of the form "Is object $x$ closer to $y$ or is $x$ closer to $z$?" In order to learn from such data, the objects are typically embedded in a Euclidean space while satisfying as many triplet comparisons as possible. In this paper, we introduce empirical uncertainty estimates for standard embedding algorithms when few noisy triplets are available, using a bootstrap and a Bayesian approach. In particular, simulations show that these estimates are well calibrated and can serve to select embedding parameters or to quantify uncertainty in scientific applications.

Motivation & Objective

• Address the lack of uncertainty quantification in ordinal embedding methods when only a small number of noisy triplets are available.
• Develop empirical uncertainty estimates that are well-calibrated and applicable to real-world scientific and psychophysical applications.
• Evaluate whether uncertainty estimates can guide embedding parameter selection, such as optimal embedding dimension.
• Assess the utility of uncertainty estimates in active learning for triplet selection.
• Provide a practical framework for quantifying confidence in ordinal embedding results under data scarcity.

Proposed method

• Apply a bootstrap approach by resampling noisy triplets to generate multiple embeddings and compute empirical variance in point positions.
• Use a Bayesian nonparametric model with Gaussian processes to infer posterior distributions over perception functions and derive uncertainty in embeddings.
• Normalize embeddings to remove scale and rotation invariance, ensuring consistent comparison across bootstrap and Bayesian samples.
• Compute mean positions and standard deviations across bootstrap samples to quantify uncertainty per data point.
• Use uncertainty estimates to rank and select the most informative triplets in active learning settings.
• Evaluate uncertainty calibration via coverage rates of prediction intervals across multiple trials.

Experimental results

Research questions

• RQ1 Can bootstrap and Bayesian methods provide well-calibrated uncertainty estimates for ordinal embeddings when only a few noisy triplets are available?
• RQ2 How effective are uncertainty estimates in guiding the selection of embedding dimension or other hyperparameters?
• RQ3 To what extent can uncertainty estimates improve active learning performance in ordinal embedding tasks?
• RQ4 How do uncertainty estimates perform in psychophysical experiments with noisy human perception data?
• RQ5 Are the uncertainty estimates robust to varying levels of noise and perceptual variability in triplet responses?

Key findings

• The bootstrap and Bayesian uncertainty estimates are well-calibrated, with prediction intervals covering true triplet outcomes at rates close to nominal levels.
• Uncertainty increases with higher perceptual variability (shorter lengthscale in GP kernel), reflecting increased inconsistency in triplet responses.
• The uncertainty estimates successfully indicate appropriate embedding dimensions when used with the bootstrap method, particularly in low-data regimes.
• Despite good calibration, uncertainty estimates showed limited improvement in active learning, with random triplet selection often outperforming uncertainty-based selection.
• The methods quantified meaningful uncertainty in psychophysics applications, where perception functions varied across observers and triplet answers were noisy.
• The framework enables reliable uncertainty quantification for both point locations and triplet predictions, even with as few as 2,000 triplets.