[Paper Review] Correlation Congruence for Knowledge Distillation

Baoyun Peng, Jin Xiao+6 more|arXiv (Cornell University)|Apr 3, 2019

Domain Adaptation and Few-Shot Learning39 references3 citations

TL;DR

This paper proposes Correlation Congruence Knowledge Distillation (CCKD), a novel knowledge distillation framework that enhances student network performance by transferring both instance-level knowledge and inter-instance correlation from teacher to student. By leveraging a Taylor-series-expanded kernel method to model correlation and optimizing mini-batch sampling strategies, CCKD achieves state-of-the-art accuracy on image classification and metric learning benchmarks, outperforming standard KD and prior SOTA methods.

ABSTRACT

Most teacher-student frameworks based on knowledge distillation (KD) depend on a strong congruent constraint on instance level. However, they usually ignore the correlation between multiple instances, which is also valuable for knowledge transfer. In this work, we propose a new framework named correlation congruence for knowledge distillation (CCKD), which transfers not only the instance-level information, but also the correlation between instances. Furthermore, a generalized kernel method based on Taylor series expansion is proposed to better capture the correlation between instances. Empirical experiments and ablation studies on image classification tasks (including CIFAR-100, ImageNet-1K) and metric learning tasks (including ReID and Face Recognition) show that the proposed CCKD substantially outperforms the original KD and achieves state-of-the-art accuracy compared with other SOTA KD-based methods. The CCKD can be easily deployed in the majority of the teacher-student framework such as KD and hint-based learning methods.

Motivation & Objective

• To address the limitation of existing knowledge distillation methods that focus only on instance-level congruence and ignore inter-instance correlations.
• To improve student network generalization by preserving the cohesive intra-class and separable inter-class structure of the teacher's embedding space.
• To develop a generalized kernel-based method for capturing complex correlations between instances in mini-batches.
• To explore effective mini-batch sampling strategies that enhance correlation knowledge transfer.
• To validate the effectiveness of CCKD across diverse vision tasks, including image classification and metric learning.

Proposed method

• Introduces Correlation Congruence Knowledge Distillation (CCKD), which jointly optimizes instance-level and correlation-level knowledge transfer between teacher and student networks.
• Employs a generalized kernel method based on Taylor series expansion of the Gaussian RBF kernel to model and capture high-order correlations between instances in a mini-batch.
• Applies a fully connected layer with equal dimensionality to align the embedding spaces of teacher and student networks, enabling direct correlation comparison.
• Proposes two novel mini-batch sampling strategies—class-uniform random sampler (CUR-sampler) and superclass-uniform random sampler (SUR-sampler)—to balance intra-class and inter-class correlation learning.
• Uses cosine similarity heatmaps to visualize and analyze the cohesion of intra-class and separation of inter-class embeddings in the student’s feature space.
• Trains the student network using a combined loss function that includes standard knowledge distillation loss and a correlation congruence loss derived from kernelized correlation matrices.

Experimental results

Research questions

• RQ1 Can preserving inter-instance correlations in the embedding space improve knowledge distillation performance beyond instance-level mimicry?
• RQ2 How does the order of Taylor series expansion in the kernel method affect the modeling of high-order correlations and downstream performance?
• RQ3 What is the impact of different mini-batch sampling strategies on the quality of correlation knowledge transfer?
• RQ4 Does CCKD lead to more cohesive intra-class and separable inter-class representations in the student network’s embedding space?
• RQ5 How does CCKD compare to state-of-the-art knowledge distillation methods across diverse vision tasks?

Key findings

• CCKD achieves state-of-the-art performance on CIFAR-100, outperforming standard KD by 1.3% in top-1 accuracy and surpassing other SOTA methods.
• On ImageNet-1K, CCKD improves top-1 accuracy by 1.1% over standard KD and achieves 74.5% top-1 accuracy with ResNet-18.
• On the MSMT17 ReID benchmark, CCKD achieves 59.6% rank-1 accuracy and 31.1% mAP with ResNet-18, significantly outperforming standard KD and other SOTA methods.
• The 3rd-order Taylor series expansion of the Gaussian RBF kernel yields the best performance, improving mAP by 0.7% over 1st-order and 0.3% over 2nd-order.
• The SUR-sampler with k=4 achieves the highest performance (59.6% rank-1, 31.1% mAP), outperforming both UR-sampler and CUR-sampler under similar settings.
• Visualization shows that CCKD produces significantly higher intra-class cosine similarities than standard KD, indicating stronger intra-class cohesion in the student’s embedding space.