[Paper Review] A Simple and Effective Framework for Pairwise Deep Metric Learning

Qi Qi, Yan Yan+2 more|arXiv (Cornell University)|Dec 24, 2019

Face and Expression Recognition27 references4 citations

TL;DR

This paper proposes a distributionally robust optimization (DRO) framework for pairwise deep metric learning that treats metric learning as a binary classification problem over pairs, addressing data imbalance through adaptive sampling. The method achieves state-of-the-art performance on benchmark datasets by optimizing sampling probabilities via DRO, offering theoretical grounding and flexibility to unify or improve existing loss functions.

ABSTRACT

Deep metric learning (DML) has received much attention in deep learning due to its wide applications in computer vision. Previous studies have focused on designing complicated losses and hard example mining methods, which are mostly heuristic and lack of theoretical understanding. In this paper, we cast DML as a simple pairwise binary classification problem that classifies a pair of examples as similar or dissimilar. It identifies the most critical issue in this problem--imbalanced data pairs. To tackle this issue, we propose a simple and effective framework to sample pairs in a batch of data for updating the model. The key to this framework is to define a robust loss for all pairs over a mini-batch of data, which is formulated by distributionally robust optimization. The flexibility in constructing the uncertainty decision set of the dual variable allows us to recover state-of-the-art complicated losses and also to induce novel variants. Empirical studies on several benchmark data sets demonstrate that our simple and effective method outperforms the state-of-the-art results. Codes are available at: https://github.com/qiqi-helloworld/A-Simple-and-Effective-Framework-for-Pairewise-Distance-Metric-Learning

Motivation & Objective

• Address the fundamental challenge of severe class imbalance between positive and negative pairs in pairwise deep metric learning.
• Provide a theoretically justified framework that unifies and improves upon existing heuristic loss functions and pair mining strategies.
• Develop a flexible, general-purpose method for DML that combines robust loss formulation with adaptive sampling for better model convergence and performance.
• Demonstrate that simple pairwise losses can achieve state-of-the-art results when combined with principled sampling via distributionally robust optimization.

Proposed method

• Formulate deep metric learning as a pairwise binary classification problem, where each pair is labeled as similar (positive) or dissimilar (negative).
• Address data imbalance by modeling the sampling distribution of pairs using distributionally robust optimization (DRO), which optimizes over an uncertainty set of the dual variable.
• Define a robust loss function over mini-batch pairs by maximizing the weighted loss under distributional uncertainty, enabling adaptive sampling of informative pairs.
• Use stochastic gradient descent with gradients computed from pairs sampled according to the optimized sampling distribution, improving training stability and performance.
• Construct uncertainty sets for the dual variable in DRO to allow recovery of existing losses (e.g., margin, MS loss) and derivation of novel variants.
• Implement three variants: DRO-TopK M (top-K sampling), DRO-TopK-PN M (top-K per positive-negative pair), and DRO-KL M (KL divergence-based uncertainty), each with distinct sampling strategies.

Experimental results

Research questions

• RQ1 Why do complex loss functions and heuristic pair mining methods in DML often fail to generalize or lack theoretical justification?
• RQ2 Can a unified framework based on distributionally robust optimization improve both performance and interpretability in pairwise deep metric learning?
• RQ3 How does adaptive sampling via DRO compare to existing pair mining methods (e.g., DWS, SH, MS) in terms of accuracy and efficiency?
• RQ4 To what extent does the proposed framework generalize to recover or improve upon state-of-the-art loss functions like MS loss or LS loss?

Key findings

• The proposed DRO-TopK-PN M variant achieves the highest recall@1 across all positive-negative pair ratios on the In-Shop dataset, outperforming DWS, SH, and other baselines.
• DRO-TopK M achieves optimal performance at K=1280 when batch size B=640, with stable performance across a wide range of K values, indicating low sensitivity to hyperparameter choice.
• The DRO framework runs faster than anchor-based mining methods like SH and DWS, and is more efficient than LS loss due to direct pair selection without per-anchor computation.
• DRO-based methods maintain or improve recall@1 as the positive-negative pair ratio increases, unlike DWS, which degrades sharply under low ratios, demonstrating robustness to imbalance.
• The framework successfully recovers or improves upon state-of-the-art losses such as MS loss and LS loss by embedding them within the DRO uncertainty set, providing theoretical grounding.
• Empirical results show that the DRO framework generalizes well across multiple benchmark datasets, achieving SOTA performance with minimal architectural or loss complexity.