[Paper Review] ProxylessKD: Direct Knowledge Distillation with Inherited Classifier for Face Recognition

Weidong Shi, Guanghui Ren+2 more|arXiv (Cornell University)|Oct 31, 2020

Face recognition and analysis29 references4 citations

TL;DR

This paper proposes ProxylessKD, a knowledge distillation method for face recognition that directly optimizes face recognition accuracy by inheriting the teacher model's classifier as the student's fixed classifier. By aligning the student's embedding space with the teacher's through this inherited classifier, ProxylessKD enables effective use of large-margin loss functions and achieves state-of-the-art performance on multiple benchmarks, outperforming prior distillation methods including L2KD.

ABSTRACT

Knowledge Distillation (KD) refers to transferring knowledge from a large model to a smaller one, which is widely used to enhance model performance in machine learning. It tries to align embedding spaces generated from the teacher and the student model (i.e. to make images corresponding to the same semantics share the same embedding across different models). In this work, we focus on its application in face recognition. We observe that existing knowledge distillation models optimize the proxy tasks that force the student to mimic the teacher's behavior, instead of directly optimizing the face recognition accuracy. Consequently, the obtained student models are not guaranteed to be optimal on the target task or able to benefit from advanced constraints, such as large margin constraints (e.g. margin-based softmax). We then propose a novel method named ProxylessKD that directly optimizes face recognition accuracy by inheriting the teacher's classifier as the student's classifier to guide the student to learn discriminative embeddings in the teacher's embedding space. The proposed ProxylessKD is very easy to implement and sufficiently generic to be extended to other tasks beyond face recognition. We conduct extensive experiments on standard face recognition benchmarks, and the results demonstrate that ProxylessKD achieves superior performance over existing knowledge distillation methods.

Motivation & Objective

• To address the limitation of existing knowledge distillation methods that optimize proxy tasks (e.g., soft label matching or activation consistency) instead of the actual face recognition accuracy.
• To enable the student model to benefit from advanced large-margin loss functions (e.g., Arcface, Cosface) that are incompatible with standard distillation due to classifier mismatch.
• To directly align the embedding spaces of student and teacher models, ensuring compatibility with real-world face recognition pipelines where gallery embeddings are extracted using large models.
• To develop a simple, generic, and effective distillation framework that does not require retraining the student’s classifier and can be easily extended beyond face recognition.

Proposed method

• The student model’s classifier is initialized with the teacher model’s classifier weights and kept fixed during training, effectively transferring the teacher’s class center representation to the student.
• The student network is trained to produce embeddings that align with the fixed, inherited classifier, thereby learning a discriminative embedding space consistent with the teacher’s.
• The method combines the student’s feature learning with a large-margin loss (e.g., Arcface or Cosface), enabling the student to benefit from margin constraints that improve inter-class separation.
• Unlike L2KD, which minimizes only the L2 distance between student and teacher embeddings, ProxylessKD jointly optimizes intra-class compactness and inter-class margin through the fixed classifier.
• The framework is generic and does not require architectural changes; it only requires weight initialization and freezing of the student’s classifier.
• The approach enables direct embedding space alignment without relying on soft labels or intermediate feature matching, making it suitable for real-world deployment where gallery features are extracted from large models.

Experimental results

Research questions

• RQ1 Can knowledge distillation in face recognition be improved by directly optimizing the target task (i.e., face recognition accuracy) rather than proxy objectives like soft label matching or feature activation consistency?
• RQ2 Why do existing distillation methods fail to benefit from large-margin loss functions such as Arcface and Cosface, and can this limitation be overcome?
• RQ3 Does inheriting the teacher’s classifier as a fixed component in the student model lead to better embedding space alignment and improved recognition performance?
• RQ4 How does ProxylessKD compare to L2KD and other distillation baselines in terms of performance across multiple standard benchmarks under both single and multiple model evaluation modes?

Key findings

• ProxylessKD achieves 95.04% verification accuracy on CFP-FP under multiple model mode, outperforming L2KD by 0.74% and demonstrating consistent gains across all benchmarks.
• On IJB-C at FAR=1e-6, ProxylessKD-m achieves 87.90% Top-1 identification accuracy, a 10.5% improvement over the student baseline and 1.03% over L2KD-m.
• On MegaFace, ProxylessKD-m achieves 95.80% Rank-1 accuracy (1.03% higher than L2KD-m), confirming its superiority on large-scale, complex recognition tasks.
• The method achieves 93.36% verification TAR at FAR=1e-6 on IJB-B, a 12.5% improvement over the student baseline and 2.5% over L2KD-s, showing strong generalization.
• ProxylessKD-m consistently outperforms ProxylessKD-s across all datasets, confirming that multi-model fusion with a large teacher model enhances performance.
• The performance gains are attributed to effective embedding space alignment and the ability to leverage large-margin constraints, which L2KD cannot utilize due to classifier mismatch.