[Paper Review] Sampling Matters in Deep Embedding Learning

Chao-Yuan Wu, R. Manmatha+2 more|arXiv (Cornell University)|Jun 23, 2017

Face recognition and analysisComputer Science38 references85 citations

TL;DR

The paper argues that training sample selection is as important as loss choice in deep embedding learning, introducing distance weighted sampling and a margin-based loss that together achieve state-of-the-art results across several benchmarks.

ABSTRACT

Deep embeddings answer one simple question: How similar are two images? Learning these embeddings is the bedrock of verification, zero-shot learning, and visual search. The most prominent approaches optimize a deep convolutional network with a suitable loss function, such as contrastive loss or triplet loss. While a rich line of work focuses solely on the loss functions, we show in this paper that selecting training examples plays an equally important role. We propose distance weighted sampling, which selects more informative and stable examples than traditional approaches. In addition, we show that a simple margin based loss is sufficient to outperform all other loss functions. We evaluate our approach on the Stanford Online Products, CAR196, and the CUB200-2011 datasets for image retrieval and clustering, and on the LFW dataset for face verification. Our method achieves state-of-the-art performance on all of them.

Motivation & Objective

• Motivate the importance of sample selection in deep embedding learning alongside loss design.
• Analyze existing sampling strategies and their impact on embedding quality.
• Propose distance weighted sampling to reduce gradient variance and stabilize training.
• Introduce a margin-based loss that adapts to data geometry and improves robustness.
• Demonstrate state-of-the-art performance across standard embedding benchmarks.

Proposed method

• Define distance D_ij as the Euclidean distance between embeddings f(x_i) and f(x_j).
• Propose distance weighted sampling with probabilities proportional to inverse distance (clipped for stability) to select negatives.
• Introduce a simple margin-based loss ell^margin(i,j) = (alpha + y_ij (D_ij - beta))_+ with class- and image-specific beta terms and a nu-regularizer for beta parameters.
• Show that the margin-based loss relaxes constraints and focuses on relative order of distances, akin to isotonic regression on distances.
• Compare sampling strategies (random, hard/ semi-hard mining) and losses (contrastive, triplet, margin) empirically on standard datasets.
• Provide analysis linking gradient variance to sample distance and how distance weighted sampling mitigates high-variance gradients.

Experimental results

Research questions

• RQ1 How does sampling strategy affect the effectiveness of different embedding losses (contrastive, triplet, margin) in deep embedding learning?
• RQ2 Can a distance-weighted sampling approach improve training stability and final embedding quality across datasets?
• RQ3 Does a margin-based loss with adaptive boundaries offer robustness and performance gains over traditional pairwise/triplet losses?
• RQ4 To what extent do sampling choices influence convergence speed and retrieval/ clustering/ verification performance?

Key findings

• Distance weighted sampling yields more informative and stable negatives than traditional sampling, reducing gradient variance and improving embedding quality.
• A simple margin-based loss with adaptive boundary beta outperforms conventional losses across multiple datasets and sampling schemes.
• Distance weighted sampling combined with the margin-based loss achieves state-of-the-art results on Stanford Online Products, CARS196, and CUB200-2011 for retrieval and clustering, and improves LFW verification performance.
• The margin-based loss, with class- and image-specific beta terms and isotonic-regression-like behavior, focuses on maintaining correct relative ordering rather than enforcing fixed margins.
• The proposed approach converges faster and more stably than triplet semi-hard mining or contrastive random sampling across evaluated settings.