[Paper Review] Relational Knowledge Distillation

Wonpyo Park, Dong‐Ju Kim+2 more|arXiv (Cornell University)|Apr 10, 2019

Machine Learning and Data Classification42 references50 citations

TL;DR

This paper introduces Relational Knowledge Distillation (RKD), which transfers mutual relations among data samples from a teacher to a student via distance-wise and angle-wise losses, improving performance across metric learning, classification, and few-shot learning.

ABSTRACT

Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher. We introduce a novel approach, dubbed relational knowledge distillation (RKD), that transfers mutual relations of data examples instead. For concrete realizations of RKD, we propose distance-wise and angle-wise distillation losses that penalize structural differences in relations. Experiments conducted on different tasks show that the proposed method improves educated student models with a significant margin. In particular for metric learning, it allows students to outperform their teachers' performance, achieving the state of the arts on standard benchmark datasets.

Motivation & Objective

• Motivate knowledge transfer from a large teacher to a smaller student to reduce computation and memory costs.
• Propose a relational perspective on KD that emphasizes structure over individual outputs.
• Introduce two concrete RKD losses (distance-wise and angle-wise) to capture high-order relations.
• Demonstrate RKD’s effectiveness across metric learning, image classification, and few-shot learning.

Proposed method

• Formalize RKD as transferring relational potentials psi over n-tuples of data using a loss between teacher and student relations.
• Define distance-wise distillation psi_D(t_i,t_j) = (1/μ) ||t_i - t_j||_2 with μ as the mini-batch average distance.
• Define angle-wise distillation psi_A(t_i,t_j,t_k) = cos(angle t_i t_j t_k) using normalized vectors between embeddings.
• Use aHuber loss l_δ to measure discrepancy between teacher and student relational potentials.
• Combine RKD losses with task-specific losses in a joint objective L_task + λ_KD L_KD with tunable weights.
• Apply RKD on embeddings from teacher and student, enabling high-order relational transfer independent of output dimensions.

Experimental results

Research questions

• RQ1 Can transferring relational information between data examples improve student models beyond conventional output-based KD?
• RQ2 Do distance-wise and angle-wise relational losses capture complementary high-order structures that aid learning?
• RQ3 Is RKD effective across diverse tasks such as metric learning, image classification, and few-shot learning?

Key findings

• RKD improves student performance across metric learning, classification, and few-shot learning compared with traditional KD baselines.
• In metric learning, RKD enables smaller students to outperform their teachers on standard benchmarks.
• Angle-wise (RKD-A) often yields faster convergence and better performance than distance-only distillation.
• RKD is complementary to other KD methods and can further boost results when combined (e.g., RKD with HKD).
• Self-distillation with RKD shows that student models can surpass their teachers across multiple datasets.