[Paper Review] Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning

Xun Wang, Xintong Han+3 more|arXiv (Cornell University)|Apr 14, 2019

Advanced Image and Video Retrieval TechniquesComputer Science6 references105 citations

TL;DR

This paper introduces a General Pair Weighting (GPW) framework for pair-based deep metric learning and proposes the Multi-Similarity (MS) loss, which jointly uses self-, positive-relative, and negative-relative similarities to mining and weight informative pairs, achieving new state-of-the-art results on several image retrieval benchmarks.

ABSTRACT

A family of loss functions built on pair-based computation have been proposed in the literature which provide a myriad of solutions for deep metric learning. In this paper, we provide a general weighting framework for understanding recent pair-based loss functions. Our contributions are three-fold: (1) we establish a General Pair Weighting (GPW) framework, which casts the sampling problem of deep metric learning into a unified view of pair weighting through gradient analysis, providing a powerful tool for understanding recent pair-based loss functions; (2) we show that with GPW, various existing pair-based methods can be compared and discussed comprehensively, with clear differences and key limitations identified; (3) we propose a new loss called multi-similarity loss (MS loss) under the GPW, which is implemented in two iterative steps (i.e., mining and weighting). This allows it to fully consider three similarities for pair weighting, providing a more principled approach for collecting and weighting informative pairs. Finally, the proposed MS loss obtains new state-of-the-art performance on four image retrieval benchmarks, where it outperforms the most recent approaches, such as ABE\cite{Kim_2018_ECCV} and HTL by a large margin: 60.6% to 65.7% on CUB200, and 80.9% to 88.0% on In-Shop Clothes Retrieval dataset at Recall@1. Code is available at https://github.com/MalongTech/research-ms-loss.

Motivation & Objective

• Cast the sampling problem of deep metric learning into a general pair weighting framework (GPW) via gradient analysis.
• Analyze and compare existing pair-based losses through GPW to identify key similarities and limitations.
• Introduce the Multi-Similarity (MS) loss with two iterative steps (mining and weighting) to utilize three types of similarities in pair weighting.
• Demonstrate state-of-the-art image retrieval performance on standard benchmarks (CUB200, Cars-196, SOP, In-Shop).

Proposed method

• Define GPW to express pair-based losses as weighted sums of pairwise similarities on a unit-sphere embedding.
• Identify three types of pair similarities: Self-similarity (S), Positive-relative similarity (P), and Negative-relative similarity (N).
• Propose an MS loss with two stages: (i) mining informative pairs based on Similarity-P, (ii) weighting mined pairs using Similarity-S and Similarity-N with equations (13) and (14).
• Provide the MS loss formulation (Eq. 15) that combines mining and weighted penalties for positive and negative pairs.
• Use two hyper-parameterized weighting components derived from binomial deviance and lifted structure concepts for robust weighting.
• Evaluate MS loss in an end-to-end PyTorch implementation with Inception backbone on four datasets (CUB200, Cars-196, SOP, In-Shop).

Experimental results

Research questions

• RQ1 Can a unified General Pair Weighting (GPW) framework explain existing pair-based losses and reveal their limitations?
• RQ2 Do multi-aspect similarities (Self, Positive-relative, Negative-relative) improve the informativeness of sampled pairs in deep metric learning?
• RQ3 Does the proposed Multi-Similarity (MS) loss outperform existing pair-based losses on standard image retrieval benchmarks?
• RQ4 How does integrating mining and weighting in MS compare to using individual components or simple combinations of prior losses?

Key findings

• MS loss with GPW achieves state-of-the-art performance on several image retrieval benchmarks.
• On CUB200, MS loss improves Recall@1 by about 5 percentage points over a recent method (ABE).
• On In-Shop, MS loss achieves Recall@1 improvements of around 7 percentage points over HTL, with evidence of strong gains on Recall@1 (e.g., 80.9% to 88.0%).
• MS loss with embedding size 128–512 consistently outperforms baselines and comparable methods across CUB200, Cars-196, SOP, and In-Shop datasets.
• Ablation studies show that incorporating Similarity-S, Similarity-P, and Similarity-N in mining and weighting yields the best Recall@K, outperforming variants that use subsets of similarities.
• MS loss remains effective across embedding sizes, with increasing gains up to 512 dimensions (diminishing returns beyond).