[论文解读] Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework

Cheng Yang, Jiawei Liu+1 more|arXiv (Cornell University)|Mar 4, 2021

Advanced Graph Neural Networks参考文献 40被引用 8

一句话总结

本文提出了一种知识蒸馏框架，将任何预训练图神经网络（GNN）教师模型的知识迁移至一个轻量级、可解释的学生模型，该学生模型由可学习的标签传播和特征转换模块组成。在五个基准数据集上，针对七种GNN架构，该学生模型在性能上超越了其教师模型，通过结合结构和基于特征的归纳偏置以及蒸馏的GNN知识，实现了1.4%至4.7%的一致性准确率提升。

ABSTRACT

Semi-supervised learning on graphs is an important problem in the machine learning area. In recent years, state-of-the-art classification methods based on graph neural networks (GNNs) have shown their superiority over traditional ones such as label propagation. However, the sophisticated architectures of these neural models will lead to a complex prediction mechanism, which could not make full use of valuable prior knowledge lying in the data, e.g., structurally correlated nodes tend to have the same class. In this paper, we propose a framework based on knowledge distillation to address the above issues. Our framework extracts the knowledge of an arbitrary learned GNN model (teacher model), and injects it into a well-designed student model. The student model is built with two simple prediction mechanisms, i.e., label propagation and feature transformation, which naturally preserves structure-based and feature-based prior knowledge, respectively. In specific, we design the student model as a trainable combination of parameterized label propagation and feature transformation modules. As a result, the learned student can benefit from both prior knowledge and the knowledge in GNN teachers for more effective predictions. Moreover, the learned student model has a more interpretable prediction process than GNNs. We conduct experiments on five public benchmark datasets and employ seven GNN models including GCN, GAT, APPNP, SAGE, SGC, GCNII and GLP as the teacher models. Experimental results show that the learned student model can consistently outperform its corresponding teacher model by 1.4% - 4.7% on average. Code and data are available at https://github.com/BUPT-GAMMA/CPF

研究动机与目标

• 为解决复杂GNN模型中结构和基于特征的归纳偏置利用不足的问题。
• 开发一种与任何预训练GNN教师模型兼容的通用知识蒸馏框架。
• 通过结合先验知识与蒸馏的GNN知识，提升半监督节点分类的准确率，超越当前最先进GNN模型。
• 通过设计具有透明、模块化组件的学生模型，增强模型的可解释性。
• 通过在低标签设置下保持性能，实现有效的少样本学习。

提出的方法

• 学生模型是可学习的参数化标签传播（LP）与用于特征转换（FT）的两层多层感知机（MLP）的可训练组合，分别编码基于结构的先验和基于特征的先验。
• 通过在学生模型输出上使用交叉熵损失，匹配GNN教师模型预测的软标签，实现知识蒸馏。
• 通过每个节点的可学习注意力类似参数α_v控制LP与FT组件之间的平衡，实现对结构与特征信息的自适应融合。
• 引入节点置信度分数c_v，用于在标签传播过程中加权每个节点的影响，提升对噪声或多样化邻域的鲁棒性。
• 使用结合知识蒸馏与标准交叉熵监督的混合损失，端到端训练学生模型。
• 在五个公开数据集上，使用七种GNN（GCN、GAT、SAGE、APPNP、SGC、GCNII、GLP）作为教师模型对框架进行评估。

Figure 1 . An overview of our knowledge distillation framework. The two simple prediction mechanisms of our student model ensure the full use of structure/feature-based prior knowledge. The knowledge in GNN teachers will be extracted and injected into the student during knowledge distillation. Thus

实验结果

研究问题

• RQ1 一个显式编码基于结构和基于特征归纳偏置的学生模型，是否能在半监督节点分类任务中超越其GNN教师模型？
• RQ2 知识蒸馏能否有效将来自多样化GNN架构的知识迁移至统一的、可解释的学生模型？
• RQ3 所提出的框架在低标签设置（少样本学习）下能多大程度上提升性能？
• RQ4 所学习的平衡参数α_v与置信度分数c_v如何反映模型的可解释性与决策过程？
• RQ5 学生模型是否能在不同GNN教师模型与数据集上实现泛化，同时保持一致的性能增益？

主要发现

• 所提出的学⽣模型在五个基准数据集上始终优于所有七种GNN教师模型，平均准确率提升1.4%至4.7%。
• 即使在低标签设置下，模型也实现了显著改进，分别在每类5、10、20和50个标记节点时，准确率提升达4.9%、4.5%、3.2%和2.1%。
• 案例研究显示，α_v值较高的节点更可能拥有预测标签相同的邻居，表明其对标签传播的强依赖。
• 置信度分数c_v较高的节点通常邻居较少且标签预测更一致，表明对邻域多样性的鲁棒性。
• α_v或c_v值较低的节点通常对应于邻居标签多样或冲突的情况，证实了模型对邻域同质性的敏感性。
• 该框架在多种GNN架构与数据集类型上均表现出强鲁棒性与泛化能力，验证了其作为通用蒸馏框架的有效性。

Figure 2 . An illustration of the architecture of our proposed student model. Taking the center node $v$ as an example, the student model starts from node $v$ ’s raw features and a uniform label distribution as soft labels. Then at each layer, the soft label prediction of $v$ will be updated as a tr