[Paper Review] S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric Learning

Karsten Roth, Timo Milbich+3 more|arXiv (Cornell University)|Sep 17, 2020

Domain Adaptation and Few-Shot Learning70 references4 citations

TL;DR

S2SD proposes a novel self-distillation framework for deep metric learning that simultaneously trains a high-dimensional auxiliary embedding space and distills its similarity knowledge into a lower-dimensional base embedding, improving generalization without increasing test-time cost. The method achieves up to 7% relative improvement in Recall@1 across standard benchmarks, setting a new state-of-the-art while maintaining minimal training overhead.

ABSTRACT

Deep Metric Learning (DML) provides a crucial tool for visual similarity and zero-shot applications by learning generalizing embedding spaces, although recent work in DML has shown strong performance saturation across training objectives. However, generalization capacity is known to scale with the embedding space dimensionality. Unfortunately, high dimensional embeddings also create higher retrieval cost for downstream applications. To remedy this, we propose \emph{Simultaneous Similarity-based Self-distillation (S2SD). S2SD extends DML with knowledge distillation from auxiliary, high-dimensional embedding and feature spaces to leverage complementary context during training while retaining test-time cost and with negligible changes to the training time. Experiments and ablations across different objectives and standard benchmarks show S2SD offers notable improvements of up to 7% in Recall@1, while also setting a new state-of-the-art. Code available at https://github.com/MLforHealth/S2SD.

Motivation & Objective

• To address performance saturation in deep metric learning despite high model capacity.
• To improve generalization in low-dimensional embedding spaces without increasing test-time retrieval cost.
• To leverage high-dimensional auxiliary representations for knowledge distillation without requiring external teacher models.
• To enable efficient, single-stage training that boosts base model performance using complementary context from higher-dimensional features.
• To explore multiscale distillation and hierarchical knowledge transfer for enhanced representation learning.

Proposed method

• Introduces a simultaneous training setup where a high-dimensional auxiliary embedding space is learned in parallel with the base embedding space using a shared feature backbone.
• Employs knowledge distillation by transferring similarity relationships from the high-dimensional auxiliary space to the lower-dimensional base space.
• Uses a distillation loss based on relative similarity rankings (e.g., using KL divergence) between pairs of samples in the auxiliary and base embedding spaces.
• Applies multiple distillation branches at varying dimensions to enable multiscale knowledge transfer and improve feature reusability.
• Implements a detachable auxiliary branch to prevent gradient interference during backpropagation while preserving distillation signal.
• Integrates distillation between the feature backbone and the base embedding space to improve feature utilization and representation quality.

Experimental results

Research questions

• RQ1 Can high-dimensional auxiliary representations improve the generalization of low-dimensional base embeddings in deep metric learning?
• RQ2 Does simultaneous self-distillation from an auxiliary embedding space yield better performance than standard distillation with external teacher models?
• RQ3 How does the choice of distillation objective (e.g., R-KL, cosine, Euclidean) affect performance and stability?
• RQ4 What is the optimal depth and width of the auxiliary network for effective knowledge transfer?
• RQ5 Does distilling from earlier or later feature layers yield better performance, and how does this vary with network depth?

Key findings

• S2SD achieves a 7% relative improvement in Recall@1 on CUB200-2011, significantly outperforming prior state-of-the-art methods.
• On CARS196, S2SD improves Recall@1 by up to 2.2% compared to baseline methods, with gains most pronounced in low-dimensional settings.
• The method sets a new state-of-the-art on all three benchmarks: CUB200-2011, CARS196, and Stanford Online Products.
• Distillation from higher-dimensional auxiliary branches (e.g., 2048-dim) yields better performance than from lower dimensions, with optimal gains at 2048-dim.
• Multiscale distillation with multiple branches (e.g., #B=4) improves performance by 0.2% over single-branch distillation, indicating benefit from diverse representations.
• Detached auxiliary branches yield superior results (66.96% R@1) compared to non-detached ones (65.34% R@1), confirming the importance of gradient isolation.