[Paper Review] Deep Metric Learning by Online Soft Mining and Class-Aware Attention

Xinshao Wang, Hua Yang+3 more|arXiv (Cornell University)|Nov 4, 2018

Video Surveillance and Tracking Methods36 references4 citations

TL;DR

This paper proposes Online Soft Mining (OSM) and Class-Aware Attention (CAA) to improve deep metric learning by assigning continuous scores to all samples in a mini-batch, enabling full utilization of useful information while reducing outlier influence. By combining OSM’s soft mining with CAA’s outlier suppression, the method achieves state-of-the-art performance on fine-grained image recognition and video-based person re-identification benchmarks, outperforming prior methods by up to 4.9% in mAP on MARS.

ABSTRACT

Deep metric learning aims to learn a deep embedding that can capture the semantic similarity of data points. Given the availability of massive training samples, deep metric learning is known to suffer from slow convergence due to a large fraction of trivial samples. Therefore, most existing methods generally resort to sample mining strategies for selecting nontrivial samples to accelerate convergence and improve performance. In this work, we identify two critical limitations of the sample mining methods, and provide solutions for both of them. First, previous mining methods assign one binary score to each sample, i.e., dropping or keeping it, so they only selects a subset of relevant samples in a mini-batch. Therefore, we propose a novel sample mining method, called Online Soft Mining (OSM), which assigns one continuous score to each sample to make use of all samples in the mini-batch. OSM learns extended manifolds that preserve useful intraclass variances by focusing on more similar positives. Second, the existing methods are easily influenced by outliers as they are generally included in the mined subset. To address this, we introduce Class-Aware Attention (CAA) that assigns little attention to abnormal data samples. Furthermore, by combining OSM and CAA, we propose a novel weighted contrastive loss to learn discriminative embeddings. Extensive experiments on two fine-grained visual categorisation datasets and two video-based person re-identification benchmarks show that our method significantly outperforms the state-of-the-art.

Motivation & Objective

• To address slow convergence in deep metric learning caused by trivial samples in large-scale datasets.
• To overcome the limitation of existing sample mining methods that discard potentially useful samples via binary (hard) mining.
• To mitigate the negative impact of outliers in mined training samples, which degrade model performance.
• To develop a generic, plug-in solution compatible with existing metric learning frameworks.
• To improve discriminative embedding learning through a novel weighted contrastive loss combining OSM and CAA.

Proposed method

• Proposes Online Soft Mining (OSM), which assigns continuous mining scores to all samples in a mini-batch instead of binary selection, enabling full utilization of intra-batch information.
• Implements soft positive mining by assigning higher scores to more similar positive pairs based on Euclidean distance in the embedding space, preserving intraclass variances.
• Introduces soft negative mining that prioritizes more difficult negatives by assigning higher scores to pairs with larger loss, generalizing hard mining.
• Develops Class-Aware Attention (CAA), which computes attention scores based on class compatibility to suppress outliers by assigning them low attention weights.
• Combines OSM and CAA into a weighted contrastive loss, where pair weights are the product of OSM scores and CAA attention scores.
• The method is generic and can be integrated into existing metric learning frameworks using contrastive, triplet, or quadruplet losses.

Experimental results

Research questions

• RQ1 Can continuous mining scores improve model convergence and performance compared to binary hard mining in deep metric learning?
• RQ2 Does preserving intraclass variance through soft positive mining lead to better generalization in fine-grained recognition tasks?
• RQ3 To what extent do outliers degrade performance in metric learning, and can they be effectively suppressed during training?
• RQ4 Can a unified framework combining soft mining and outlier attention achieve state-of-the-art results across diverse benchmark datasets?
• RQ5 How does the integration of OSM and CAA affect performance on both image-based and video-based person re-identification tasks?

Key findings

• On the CUB-200-2011 dataset, the proposed method achieves 77.5% CMC-1 accuracy, outperforming the baseline by 5.5 percentage points.
• On the MARS dataset, the method improves mAP by 4.9% and CMC-1 by 2.3% compared to the state-of-the-art CAE method.
• On the more challenging LPW dataset, the method achieves 84.7% CMC-1, significantly outperforming all compared methods.
• The ablation study confirms that OSM alone improves performance by 1-2% over the baseline, and adding CAA yields an additional 1-2% gain.
• Even with pre-trained GoogleNet weights, the baseline model achieves 47.8% CMC-1, but the proposed method exceeds this by over 7 percentage points on CUB-200-2011.
• The integration of OSM and CAA leads to consistent improvements across all datasets, demonstrating robustness and generalization.