[Paper Review] Learning Deep Embeddings with Histogram Loss

Evgeniya Ustinova, Victor Lempitsky|arXiv (Cornell University)|Nov 2, 2016

Human Pose and Action RecognitionComputer Science30 references260 citations

TL;DR

The paper introduces a parameter-free Histogram Loss for learning deep embeddings by estimating distributions of positive and negative pair similarities in-batch using histograms, and minimizes their overlap via backpropagation. It achieves competitive or superior performance across multiple datasets without tuning margins or thresholds.

ABSTRACT

We suggest a loss for learning deep embeddings. The new loss does not introduce parameters that need to be tuned and results in very good embeddings across a range of datasets and problems. The loss is computed by estimating two distribution of similarities for positive (matching) and negative (non-matching) sample pairs, and then computing the probability of a positive pair to have a lower similarity score than a negative pair based on the estimated similarity distributions. We show that such operations can be performed in a simple and piecewise-differentiable manner using 1D histograms with soft assignment operations. This makes the proposed loss suitable for learning deep embeddings using stochastic optimization. In the experiments, the new loss performs favourably compared to recently proposed alternatives.

Motivation & Objective

• Motivate a robust embedding loss that minimizes dependence on tunable margins or thresholds.
• Propose a two-stage, histogram-based approach to estimate positive and negative similarity distributions.
• Enable differentiable optimization of the embedding via backpropagation through histogram-based loss.
• Demonstrate broad applicability by evaluating on diverse image embedding datasets and comparing to state-of-the-art losses.

Proposed method

• Estimate two distributions of similarities for positive and negative pairs within a batch using 1D histograms with linear interpolation.
• Compute the probability that a random negative pair has higher similarity than a random positive pair as an integral over the estimated distributions.
• Define the histogram loss L as the expected reverse probability, computable as L = sum_r h^{-}_r phi^{+}_r where phi^{+}_r is the cumulative sum of h^{+}.
• Backpropagate through the histogram construction to obtain gradients with respect to s_{ij} and hence to network parameters.
• Treat the histogram bin count as the only tunable parameter, showing low sensitivity and independence from data.

Experimental results

Research questions

• RQ1 Can a parameter-free histogram-based loss effectively separate positive and negative pairs in embedding spaces across varied datasets?
• RQ2 How does histogram size (bin count) and batch size affect embedding quality and convergence?
• RQ3 Does the Histogram Loss outperform state-of-the-art pairwise and triplet/quadruplet losses on standard embedding benchmarks?
• RQ4 Is the approach scalable to large datasets while maintaining differentiability and training efficiency?

Key findings

• Histogram Loss achieves competitive recall@K compared to Binomial Deviance, LSSS, and Triplet losses across CUB-200-2011, Online Products, CUHK03, and Market-1501 datasets.
• On CUHK03 and Market-1501, Histogram Loss outperforms competing losses for person re-identification.
• For CUB-200-2011 and Online Products, Histogram Loss comes very close to the best Binomial Deviance results (with appropriate C parameter).
• Histogram Loss uniformly outperformed the lifted structured softmax (LSSS) triplet-based loss in the evaluated datasets.
• Larger batch sizes (e.g., 256) generally yield better Recall@K across several datasets.
• The method requires minimal tuning beyond histogram bin count and standard optimization settings (e.g., ADAM).