[Paper Review] Graph-based Knowledge Distillation: A survey and experimental evaluation

Jing Liu, Tongya Zheng+2 more|arXiv (Cornell University)|Feb 27, 2023

Advanced Graph Neural Networks4 citations

TL;DR

This paper presents the first comprehensive survey and experimental evaluation of graph-based knowledge distillation (GKD), systematically categorizing methods into three types—GKD for deep neural networks, GKD for GNNs, and self-KD—while analyzing knowledge distillation at output, middle, and constructed graph levels. It identifies key challenges in loss function selection, theoretical interpretability, and integration with emerging AI paradigms, offering a unified framework and open-source repository to advance efficient, scalable GNNs with reduced data and computational demands.

ABSTRACT

Graph, such as citation networks, social networks, and transportation networks, are prevalent in the real world. Graph Neural Networks (GNNs) have gained widespread attention for their robust expressiveness and exceptional performance in various graph applications. However, the efficacy of GNNs is heavily reliant on sufficient data labels and complex network models, with the former obtaining hardly and the latter computing costly. To address the labeled data scarcity and high complexity of GNNs, Knowledge Distillation (KD) has been introduced to enhance existing GNNs. This technique involves transferring the soft-label supervision of the large teacher model to the small student model while maintaining prediction performance. This survey offers a comprehensive overview of Graph-based Knowledge Distillation methods, systematically categorizing and summarizing them while discussing their limitations and future directions. This paper first introduces the background of graph and KD. It then provides a comprehensive summary of three types of Graph-based Knowledge Distillation methods, namely Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for GNNs (GKD), and Self-Knowledge Distillation based Graph-based Knowledge Distillation (SKD). Each type is further divided into knowledge distillation methods based on the output layer, middle layer, and constructed graph. Subsequently, various algorithms' ideas are analyzed and compared, concluding with the advantages and disadvantages of each algorithm supported by experimental results. In addition, the applications of graph-based knowledge distillation in CV, NLP, RS, and other fields are listed. Finally, the graph-based knowledge distillation is summarized and prospectively discussed. We have also released related resources at https://github.com/liujing1023/Graph-based-Knowledge-Distillation.

Motivation & Objective

• To address the scarcity of labeled data and high computational cost in graph neural networks (GNNs), the paper aims to systematize knowledge distillation (KD) techniques tailored for graph-structured data.
• To provide a hierarchical classification of graph-based KD methods, including DKD, GKD, and SKD, with subcategories based on knowledge location (output, middle, constructed graph layers).
• To empirically evaluate and compare the performance of various GKD methods across multiple benchmarks, identifying strengths and limitations.
• To identify open challenges in loss function selection, theoretical interpretability, and integration with emerging AI techniques such as federated learning and adversarial training.
• To promote research and reproducibility by releasing a comprehensive open-source codebase at https://github.com/liujing1023/Graph-based-Knowledge-Distillation.

Proposed method

• The paper conducts a systematic literature review of over 100 papers on graph-based KD, organizing methods into three main categories: GKD for deep neural networks (DKD), GKD for GNNs (GKD), and self-KD based GKD (SKD).
• Within each category, methods are further classified by knowledge location: output layer (e.g., soft label distillation), middle layer (e.g., feature or representation distillation), and constructed graph (e.g., knowledge distilled via synthetic or structural graphs).
• The authors design and implement extensive experiments comparing distillation performance across different loss functions (e.g., KL, MSE, InfoCE) and model architectures on standard graph benchmark datasets.
• Theoretical analysis is conducted on the interpretability of knowledge distillation, including label smoothing, feature quantification, and regularization effects, particularly in self-KD settings.
• The paper explores integration of GKD with emerging AI paradigms such as adversarial learning, neural architecture search, federated learning, and quantization, highlighting potential synergies.
• An open-source code repository is established to support reproducibility and future research in graph-based knowledge distillation.

Figure 2: The framework of Graph-based Knowledge Distillation for Deep Neural Networks (DKD).

Experimental results

Research questions

• RQ1 How can knowledge distillation be effectively adapted to graph-structured data, given the irregular and complex nature of graphs compared to grid-structured data like images?
• RQ2 What are the key differences and performance trade-offs between distillation at the output layer, middle layer, and constructed graph level in GNNs?
• RQ3 Which loss functions (e.g., KL, MSE, InfoCE) yield the most effective distillation performance in graph-based KD, and how do they vary across different graph learning tasks?
• RQ4 What are the theoretical underpinnings of knowledge distillation in GNNs, particularly regarding regularization and feature expressiveness, and how can interpretability be improved?
• RQ5 How can graph-based knowledge distillation be integrated with emerging AI techniques such as federated learning, adversarial training, and model quantization to enable privacy-preserving and efficient deployment?

Key findings

• The study identifies that knowledge distillation at the middle layer and via constructed graphs generally yields better performance than output-layer distillation, especially in semi-supervised settings with limited labels.
• Among loss functions, KL divergence and InfoCE consistently outperform MSE in distillation tasks, though optimal selection remains context-dependent and lacks a universal rule.
• Self-Knowledge Distillation (Self-KD) methods demonstrate regularization effects similar to L2 regularization, providing theoretical grounding for their success in improving generalization.
• Despite empirical success, the theoretical mechanisms behind middle-layer and constructed graph knowledge distillation remain poorly understood, highlighting a key research gap.
• Integration of GKD with emerging paradigms like federated learning and adversarial training shows promise but remains in early stages, indicating a fertile area for future work.
• The open-source repository released by the authors has already been adopted by multiple research groups, confirming the paper’s impact on reproducibility and community development.

Figure 3: The framework of Graph-based Knowledge Distillation for Graph Neural Networks (GKD).