[论文解读] T2-GNN: Graph Neural Networks for Graphs with Incomplete Features and Structure via Teacher-Student Distillation

Cuiying Huo, Di Jin+4 more|arXiv (Cornell University)|Dec 24, 2022

Advanced Graph Neural Networks被引用 4

一句话总结

该论文提出T2-GNN，一种教师-学生蒸馏框架，通过分别训练特征和结构教师模型，实现对学生GNN的双重蒸馏，从而在节点特征和图结构均不完整的情况下提升图神经网络（GNNs）的性能。该方法在不同掩码比率下，相较于GCN在多个数据集上实现了平均8.69%的性能提升，证明了其在处理不完整特征与结构之间相互干扰问题上的鲁棒性与有效性。

ABSTRACT

Graph Neural Networks (GNNs) have been a prevailing technique for tackling various analysis tasks on graph data. A key premise for the remarkable performance of GNNs relies on complete and trustworthy initial graph descriptions (i.e., node features and graph structure), which is often not satisfied since real-world graphs are often incomplete due to various unavoidable factors. In particular, GNNs face greater challenges when both node features and graph structure are incomplete at the same time. The existing methods either focus on feature completion or structure completion. They usually rely on the matching relationship between features and structure, or employ joint learning of node representation and feature (or structure) completion in the hope of achieving mutual benefit. However, recent studies confirm that the mutual interference between features and structure leads to the degradation of GNN performance. When both features and structure are incomplete, the mismatch between features and structure caused by the missing randomness exacerbates the interference between the two, which may trigger incorrect completions that negatively affect node representation. To this end, in this paper we propose a general GNN framework based on teacher-student distillation to improve the performance of GNNs on incomplete graphs, namely T2-GNN. To avoid the interference between features and structure, we separately design feature-level and structure-level teacher models to provide targeted guidance for student model (base GNNs, such as GCN) through distillation. Then we design two personalized methods to obtain well-trained feature and structure teachers. To ensure that the knowledge of the teacher model is comprehensively and effectively distilled to the student model, we further propose a dual distillation mode to enable the student to acquire as much expert knowledge as possible.

研究动机与目标

• 解决由于数据缺失导致节点特征与图结构均不完整时，GNN性能下降的挑战。
• 克服现有联合补全方法中不完整特征与结构之间相互干扰所导致的GNN性能下降问题。
• 开发一种可泛化的GNN框架，通过知识蒸馏解耦特征与结构补全，避免误差传播。
• 通过在logits和中间表示层面的双重蒸馏，确保从专家教师到学生模型的全面且有效的知识迁移。
• 在具有不同特征与结构不完整程度的真实世界图上，验证方法的鲁棒性与有效性。

提出的方法

• 设计两个独立的教师模型：基于特征的教师模型在掩码节点特征上进行训练，基于结构的教师模型在重构的图结构上进行训练，以提供针对性指导。
• 使用基于PPR的邻居选择与自监督对比学习等个性化方法训练特征与结构教师模型，以增强鲁棒性。
• 实施双重蒸馏机制，通过logits蒸馏与中间表示蒸馏，将来自两个教师的知识迁移至学生GNN。
• 利用可学习超参数λ平衡来自特征与结构教师的蒸馏损失，根据数据可信度自适应调整指导权重。
• 在蒸馏损失中引入温度参数ρ，以控制对负标签的关注程度，研究表明小ρ值可降低对噪声的敏感性。
• 将框架应用于基础GNN（如GCN）作为学生模型，实现端到端训练，同时保留学生模型原有的架构与归纳偏置。

实验结果

研究问题

• RQ1 知识蒸馏能否有效缓解GNN中不完整特征与结构之间的相互干扰？
• RQ2 通过独立教师模型解耦特征与结构补全，是否优于联合学习方法？
• RQ3 与单模式蒸馏相比，logits与中间层的双重蒸馏如何提升知识迁移效果？
• RQ4 top-k邻居数、蒸馏温度ρ与平衡权重λ等超参数对模型鲁棒性有何影响？
• RQ5 T2-GNN在具有不同特征与结构不完整程度的真实世界图上，泛化能力如何？

主要发现

• 在八组基准数据集上，T2-GNN在不同掩码比率下相较GCN实现了平均8.69%的性能提升，增益范围为8.01%至9.43%。
• 消融实验表明，特征教师与结构教师均不可或缺，w/o_Teacher_str与w/o_Teacher_fea在不同数据集上表现不一致，凸显了自适应加权的必要性。
• 双重蒸馏（logits与中间表示）始终优于单模式蒸馏，证明全面的知识迁移可显著提升学生模型性能。
• 最优λ值（平衡参数）因数据集而异——在Wisconsin数据集上λ=0.8，在Squirrel数据集上λ=0.4，表明不同图中特征与结构的完整程度可信度不同。
• 蒸馏温度ρ相对不敏感，小值（1–2）略优，表明减少负标签带来的噪声可提升模型鲁棒性。
• 超参数敏感性分析显示，top-k邻居选择（k）存在最优范围：邻居过少或过多均会降低性能，表明信息丰富度与噪声之间存在权衡。