[Paper Review] Triplet Distillation for Deep Face Recognition

Yushu Feng, Huan Wang+2 more|arXiv (Cornell University)|May 11, 2019

Face recognition and analysis32 references4 citations

TL;DR

This paper proposes triplet distillation, a knowledge distillation-based method that uses a teacher model to dynamically adapt margins in triplet loss for improved face recognition in lightweight student networks. By transferring inter-identity similarity knowledge from the teacher’s feature distances, the method achieves state-of-the-art performance on LFW, AgeDB, and CPLFW, outperforming fixed-margin triplet loss with a 0.52% accuracy gain on CPLFW and 0.74% on AgeDB-30.

ABSTRACT

Convolutional neural networks (CNNs) have achieved a great success in face recognition, which unfortunately comes at the cost of massive computation and storage consumption. Many compact face recognition networks are thus proposed to resolve this problem. Triplet loss is effective to further improve the performance of those compact models. However, it normally employs a fixed margin to all the samples, which neglects the informative similarity structures between different identities. In this paper, we propose an enhanced version of triplet loss, named triplet distillation, which exploits the capability of a teacher model to transfer the similarity information to a small model by adaptively varying the margin between positive and negative pairs. Experiments on LFW, AgeDB, and CPLFW datasets show the merits of our method compared to the original triplet loss.

Motivation & Objective

• To address the performance degradation in compact face recognition models due to reduced model capacity.
• To overcome the limitation of fixed-margin triplet loss, which fails to capture identity-specific similarity structures.
• To improve metric learning in lightweight models by leveraging knowledge distillation with adaptive margins.
• To transfer similarity information between identities from a large teacher model to a small student model for better decision boundary learning.

Proposed method

• The method uses a pre-trained ResNet-100 as the teacher model and a slimmed MobileFaceNet as the student model.
• It computes inter-identity distances using the teacher model’s feature embeddings to determine dynamic margins.
• The margin for each triplet is defined as a linear function of the teacher’s distance difference between positive and negative pairs: $ \mathcal{F}(d) = \frac{m_{\text{max}} - m_{\text{min}}}{d_{\text{max}}} d + m_{\text{min}} $.
• The student model is first pre-trained with ArcFace loss, then fine-tuned using the proposed triplet distillation loss with adaptive margins.
• The loss function combines the student’s feature distances with the teacher’s similarity knowledge, enforcing better intra-class compactness and inter-class separation.
• The dynamic margin ensures that harder identity pairs (with larger teacher-estimated distances) receive larger margins, improving generalization.

Experimental results

Research questions

• RQ1 Can knowledge distillation be effectively applied to improve triplet loss in face recognition by transferring similarity knowledge from a teacher model?
• RQ2 Does using dynamic, data-dependent margins based on teacher model distances lead to better performance than fixed-margin triplet loss in lightweight models?
• RQ3 Can a student model achieve higher accuracy on standard benchmarks by learning from the teacher’s inter-identity similarity structure?
• RQ4 How does the proposed triplet distillation compare to standard triplet loss with fixed margins in terms of verification accuracy across diverse datasets?

Key findings

• The student model fine-tuned with triplet distillation achieved 99.27% accuracy on LFW, outperforming the best fixed-margin triplet loss (99.23% with margin 0.4).
• On CPLFW, the method reached 81.28% accuracy, a 2.75% improvement over the best fixed-margin triplet loss (80.80% with margin 0.3).
• On AgeDB-30, the method achieved 94.37% accuracy, a 0.84% improvement over the best fixed-margin triplet loss (94.08% with margin 0.3).
• The performance gain was consistent across all three benchmarks, demonstrating the effectiveness of dynamic margins derived from teacher model knowledge.
• The student model, initially achieving 78.53% on CPLFW with ArcFace, improved to 81.28% with triplet distillation, showing strong fine-tuning capability.
• The method effectively transfers similarity structure knowledge from the teacher model, enabling better decision boundary learning in the student network.