[Paper Review] Distilling Knowledge from Graph Convolutional Networks

Yiding Yang, Jiayan Qiu+3 more|arXiv (Cornell University)|Mar 23, 2020

Advanced Neural Network ApplicationsComputer Science48 references21 citations

TL;DR

This paper proposes the first dedicated knowledge distillation method for graph convolutional networks (GCNs), introducing a Local Structure Preserving (LSP) module that transfers topological semantics from a teacher GCN to a student model by matching distributions of local node structures. The approach achieves state-of-the-art performance across multiple GCN architectures and datasets, including 91.9% accuracy on ModelNet40 with a student model using only 1/18th the parameters of the teacher.

ABSTRACT

Existing knowledge distillation methods focus on convolutional neural networks (CNNs), where the input samples like images lie in a grid domain, and have largely overlooked graph convolutional networks (GCN) that handle non-grid data. In this paper, we propose to our best knowledge the first dedicated approach to distilling knowledge from a pre-trained GCN model. To enable the knowledge transfer from the teacher GCN to the student, we propose a local structure preserving module that explicitly accounts for the topological semantics of the teacher. In this module, the local structure information from both the teacher and the student are extracted as distributions, and hence minimizing the distance between these distributions enables topology-aware knowledge transfer from the teacher, yielding a compact yet high-performance student model. Moreover, the proposed approach is readily extendable to dynamic graph models, where the input graphs for the teacher and the student may differ. We evaluate the proposed method on two different datasets using GCN models of different architectures, and demonstrate that our method achieves the state-of-the-art knowledge distillation performance for GCN models. Code is publicly available at https://github.com/ihollywhy/DistillGCN.PyTorch.

Motivation & Objective

• To address the gap in knowledge distillation for graph convolutional networks (GCNs), which are underexplored despite their growing use in non-grid data like point clouds and molecular structures.
• To enable effective knowledge transfer from a pre-trained teacher GCN to a smaller, more efficient student GCN while preserving the topological structure of the input graph.
• To design a method that explicitly accounts for the topological semantics embedded in GCN representations, going beyond standard output or activation distillation.
• To extend the approach to dynamic graph models where the graph structure may differ between teacher and student.
• To demonstrate generalization across different GCN architectures and real-world datasets in node classification and 3D object recognition.

Proposed method

• The Local Structure Preserving (LSP) module computes a distribution over the similarities between each node and its neighbors in the feature space, representing the local structure of the graph.
• The LSP module minimizes the distance between the local structure distributions of the teacher and student GCNs using a kernel-based divergence measure (e.g., RBF kernel), enabling topology-aware knowledge transfer.
• The method is formulated as a distillation loss that encourages the student model to learn a similar local structural organization as the teacher, even when the graph structures differ.
• The approach is compatible with dynamic graph models, where the graph is constructed during inference or training, not fixed in advance.
• The LSP loss is combined with standard cross-entropy loss during training, allowing end-to-end optimization of the student model.
• The method is implemented in PyTorch and publicly released, supporting various GCN architectures and hyperparameter configurations.

Experimental results

Research questions

• RQ1 Can knowledge distillation be effectively applied to graph convolutional networks (GCNs), which process non-grid, topologically structured data?
• RQ2 How can topological semantics—beyond node predictions or intermediate activations—be distilled from a teacher GCN to a student model?
• RQ3 Can a student GCN achieve performance close to the teacher while being significantly smaller in size, especially in terms of layers, channels, and graph complexity?
• RQ4 Does preserving local structural distributions between teacher and student lead to better generalization and robustness across different GCN architectures and datasets?
• RQ5 Can the distillation method be extended to dynamic graph models where the input graph structure varies between teacher and student?

Key findings

• On the ModelNet40 3D object recognition dataset, the proposed method achieved 91.9% test accuracy with a student model using only 0.1M parameters, compared to 1.81M for the teacher.
• The student model trained with the LSP module outperformed all baselines, including KD, AT, and FitNet, achieving 88.6% mean class accuracy, compared to 88.1% for KD and 87.9% for AT.
• The RBF kernel function yielded the best performance among all tested kernel functions, achieving 91.9% accuracy and 88.6% mean class accuracy on ModelNet40.
• Even with a student model having only 1/18th the number of parameters of the teacher, the LSP method achieved comparable performance, demonstrating strong parameter efficiency.
• The ablation study showed that increasing model capacity (e.g., more channels or layers) in the student model led to higher accuracy (up to 92.3% with more channels), confirming the method’s scalability and effectiveness.
• Visualization confirmed that the student model trained with LSP quickly learned a feature space structure similar to the teacher’s, especially in early training, indicating effective topological knowledge transfer.