[Paper Review] Revisiting Training Strategies and Generalization Performance in Deep Metric Learning

Karsten Roth, Timo Milbich+4 more|arXiv (Cornell University)|Feb 19, 2020

Face recognition and analysis68 references79 citations

TL;DR

This paper provides a thorough, unified evaluation of deep metric learning (DML) baselines under consistent training conditions, analyzes mini-batch sampling and embedding space compression, and introduces a simple regularization to boost generalization for ranking-based DML methods.

ABSTRACT

Deep Metric Learning (DML) is arguably one of the most influential lines of research for learning visual similarities with many proposed approaches every year. Although the field benefits from the rapid progress, the divergence in training protocols, architectures, and parameter choices make an unbiased comparison difficult. To provide a consistent reference point, we revisit the most widely used DML objective functions and conduct a study of the crucial parameter choices as well as the commonly neglected mini-batch sampling process. Under consistent comparison, DML objectives show much higher saturation than indicated by literature. Further based on our analysis, we uncover a correlation between the embedding space density and compression to the generalization performance of DML models. Exploiting these insights, we propose a simple, yet effective, training regularization to reliably boost the performance of ranking-based DML models on various standard benchmark datasets. Code and a publicly accessible WandB-repo are available at https://github.com/Confusezius/Revisiting_Deep_Metric_Learning_PyTorch.

Motivation & Objective

• Assess how training objectives, data sampling, and model design choices affect DML performance under uniform conditions.
• Evaluate the impact of mini-batch sampling on DML optimization and generalization.
• Investigate how embedding space structure (compression, margins, intra-class variance) relates to generalization in DML.
• Propose a regularization technique to improve generalization of ranking-based DML methods.

Proposed method

• Review and re-implement common DML objectives (ranking-based, classification-based, proxy-based) under consistent training settings.
• Systematically study mini-batch sampling strategies (label-based SPC n and embedded sampling with memory banks) to measure impact on performance.
• Analyze the learned embedding space via spectral decay (SVD) and class-conditional statistics to correlate generalization with embedding space properties.
• Propose and validate a regularization technique that reduces embedding space compression to boost generalization.
• Provide exhaustive experiments on standard benchmarks (CUB200-2011, CARS196, SOP) with fixed architectures (ResNet50-based) and training protocols for fair comparisons.

Experimental results

Research questions

• RQ1 How do different DML objective functions perform when training conditions are held constant across architectures and datasets?
• RQ2 What is the role of mini-batch sampling in shaping DML optimization and generalization?
• RQ3 How does the spectral decay of learned embeddings relate to generalization under train/test distribution shifts?
• RQ4 Can a regularization that controls embedding space compression improve generalization for ranking-based DML methods?

Key findings

• DML objective performance saturates under carefully matched training conditions, with many losses performing similarly on standard benchmarks.
• Batch size, data preprocessing, and architecture choice significantly influence results; inconsistent reporting hinders fair comparisons.
• There is a strong inverse correlation between generalization and embedding space spectral decay; more directions of variance support better transfer/generalization.
• Embedding space compression correlates with poorer generalization, while a simple regularization to reduce compression improves ranking-based DML performance.
• Simple mini-batch strategies (SPC-2) and larger batch diversity can outperform more complex mining methods across datasets.