[Paper Review] Deep geometric knowledge distillation with graphs

Carlos Lassance, Myriam Bontonou+4 more|arXiv (Cornell University)|Nov 8, 2019

Advanced Neural Network Applications31 references4 citations

TL;DR

This paper proposes Graph Knowledge Distillation (GKD), a dimension-agnostic knowledge distillation method that uses graph-based representations of intermediate feature activations to transfer relational knowledge from a teacher to a student network. By modeling inter-sample relationships via similarity-based graphs and minimizing adjacency matrix discrepancies, GKD achieves significantly improved student accuracy—up to twice the gain over standard RKD—on vision benchmarks without requiring matching latent space dimensions.

ABSTRACT

In most cases deep learning architectures are trained disregarding the amount of operations and energy consumption. However, some applications, like embedded systems, can be resource-constrained during inference. A popular approach to reduce the size of a deep learning architecture consists in distilling knowledge from a bigger network (teacher) to a smaller one (student). Directly training the student to mimic the teacher representation can be effective, but it requires that both share the same latent space dimensions. In this work, we focus instead on relative knowledge distillation (RKD), which considers the geometry of the respective latent spaces, allowing for dimension-agnostic transfer of knowledge. Specifically we introduce a graph-based RKD method, in which graphs are used to capture the geometry of latent spaces. Using classical computer vision benchmarks, we demonstrate the ability of the proposed method to efficiently distillate knowledge from the teacher to the student, leading to better accuracy for the same budget as compared to existing RKD alternatives.

Motivation & Objective

• Address the limitation of individual knowledge distillation (IKD), which requires matching latent space dimensions between teacher and student networks.
• Overcome the dimensionality mismatch in layer-wise distillation by enabling knowledge transfer independent of feature dimensionality.
• Improve upon existing relational knowledge distillation (RKD) by leveraging graph structures to better capture the geometric relationships in latent representations.
• Demonstrate that graph-based distillation leads to more consistent and accurate student models compared to standard RKD and baseline methods.

Proposed method

• Construct graphs where nodes represent training samples and edge weights encode the similarity between their intermediate feature representations at a given network layer.
• Use a k-nearest neighbors approach to define local graph neighborhoods, focusing on the most relevant inter-sample relationships to reduce noise.
• Define a discrepancy loss between the adjacency matrices of the teacher and student graphs to align their geometric structures during training.
• Apply the graph-based distillation loss in a layer-wise fashion, enabling dimension-agnostic knowledge transfer across architectures with different depths, widths, or input resolutions.
• Utilize graph signal smoothness analysis (via Laplacian matrices) to evaluate how well student representations align with the teacher’s geometric structure.
• Explore variations such as focusing on inter-class vs. intra-class pairs, and higher-order adjacency matrices (p > 1), to assess robustness and effectiveness.

Experimental results

Research questions

• RQ1 Can graph-based modeling of inter-sample relationships in latent spaces improve knowledge distillation beyond standard relational distillation (RKD)?
• RQ2 Does using similarity-based graphs enable effective, dimension-agnostic distillation between teacher and student networks with mismatched feature dimensions?
• RQ3 How does focusing on local neighborhoods (via k-NN) or inter-class margins affect distillation performance compared to using all pairwise distances?
• RQ4 To what extent do higher-order geometric relations (via powers of adjacency matrices) improve knowledge transfer, or do they introduce noise?
• RQ5 How well do the geometric structures of student networks align with those of the teacher, as measured by graph signal smoothness?

Key findings

• GKD achieves a test error rate of 9.43% on CIFAR-100 when k=5, outperforming the baseline RKD-D with 9.70% error, representing a ~2.8% relative improvement.
• Focusing on inter-class pairs (distinct classes) reduces error to 9.54%, indicating that margin preservation is more beneficial than clustering alignment.
• Using only intra-class pairs increases error to 10.35%, confirming that clustering similarity is less effective than margin-based alignment.
• Lowering k (e.g., to 5) improves performance over larger neighborhoods, showing that local, high-similarity relationships are more informative than global distances.
• Higher-order adjacency matrices (p=2,3) degrade performance (error rates 10.44% and 10.37%), suggesting that noise from distant or less relevant pairs harms knowledge transfer.
• Spectral analysis confirms that GKD produces smoother graph signals (e.g., label indicators and Fiedler vectors), indicating better alignment with the teacher’s latent geometry.