[Paper Review] T2-GNN: Graph Neural Networks for Graphs with Incomplete Features and Structure via Teacher-Student Distillation

Cuiying Huo, Di Jin+4 more|arXiv (Cornell University)|Dec 24, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes T2-GNN, a teacher-student distillation framework that improves Graph Neural Networks (GNNs) on graphs with incomplete features and structure by training separate feature and structure teacher models to guide a student GNN via dual distillation. The method achieves significant performance gains, with average improvements of 8.69% over GCN across multiple datasets under varying masking ratios, demonstrating robustness and effectiveness in handling mutual interference between incomplete features and structure.

ABSTRACT

Graph Neural Networks (GNNs) have been a prevailing technique for tackling various analysis tasks on graph data. A key premise for the remarkable performance of GNNs relies on complete and trustworthy initial graph descriptions (i.e., node features and graph structure), which is often not satisfied since real-world graphs are often incomplete due to various unavoidable factors. In particular, GNNs face greater challenges when both node features and graph structure are incomplete at the same time. The existing methods either focus on feature completion or structure completion. They usually rely on the matching relationship between features and structure, or employ joint learning of node representation and feature (or structure) completion in the hope of achieving mutual benefit. However, recent studies confirm that the mutual interference between features and structure leads to the degradation of GNN performance. When both features and structure are incomplete, the mismatch between features and structure caused by the missing randomness exacerbates the interference between the two, which may trigger incorrect completions that negatively affect node representation. To this end, in this paper we propose a general GNN framework based on teacher-student distillation to improve the performance of GNNs on incomplete graphs, namely T2-GNN. To avoid the interference between features and structure, we separately design feature-level and structure-level teacher models to provide targeted guidance for student model (base GNNs, such as GCN) through distillation. Then we design two personalized methods to obtain well-trained feature and structure teachers. To ensure that the knowledge of the teacher model is comprehensively and effectively distilled to the student model, we further propose a dual distillation mode to enable the student to acquire as much expert knowledge as possible.

Motivation & Objective

• Address the challenge of performance degradation in GNNs when both node features and graph structure are incomplete due to missing data.
• Overcome the mutual interference between incomplete features and structure that degrades GNN performance in existing joint completion methods.
• Develop a generalizable GNN framework that decouples feature and structure completion using knowledge distillation to avoid error propagation.
• Ensure comprehensive and effective knowledge transfer from expert teachers to the student model through dual distillation at both logits and intermediate representations.
• Demonstrate robustness and effectiveness across diverse real-world graphs with varying levels of feature and structure incompleteness.

Proposed method

• Design two separate teacher models: a feature-based teacher trained on masked node features and a structure-based teacher trained on reconstructed graph structures to provide targeted guidance.
• Train the feature and structure teachers using personalized methods that leverage PPR-based neighbor selection and self-supervised contrastive learning to enhance robustness.
• Implement a dual distillation mechanism that transfers knowledge from both teachers to the student GNN via logit distillation and intermediate representation distillation.
• Balance the distillation loss from feature and structure teachers using a learnable hyperparameter λ to adaptively weight guidance based on data trustworthiness.
• Use a temperature parameter ρ in the distillation loss to control attention to negative labels, with small ρ values shown to reduce noise sensitivity.
• Apply the framework to a base GNN (e.g., GCN) as the student model, enabling end-to-end training while preserving the student’s original architecture and inductive bias.

Experimental results

Research questions

• RQ1 Can knowledge distillation effectively mitigate the mutual interference between incomplete features and structure in GNNs?
• RQ2 Does decoupling feature and structure completion through separate teacher models lead to better performance than joint learning approaches?
• RQ3 How does dual distillation (at logits and intermediate layers) improve knowledge transfer compared to single-mode distillation?
• RQ4 What is the impact of hyperparameters such as top-k neighbors, distillation temperature ρ, and balance weight λ on model robustness?
• RQ5 To what extent does T2-GNN generalize across diverse real-world graphs with varying degrees of feature and structure incompleteness?

Key findings

• T2-GNN achieves an average performance improvement of 8.69% over GCN across eight benchmark datasets under varying masking ratios, with gains ranging from 8.01% to 9.43%.
• The ablation study confirms that both feature and structure teachers are essential, with w/o_Teacher_str and w/o_Teacher_fea showing inconsistent performance depending on the dataset, highlighting the need for adaptive weighting.
• Dual distillation (logits and intermediate representations) consistently outperforms single-mode distillation, proving that comprehensive knowledge transfer enhances student performance.
• The optimal value of λ (balance parameter) varies by dataset—λ = 0.8 on Wisconsin and λ = 0.4 on Squirrel—indicating that feature and structure completeness have different trustworthiness levels in different graphs.
• The distillation temperature ρ is relatively insensitive, with small values (1–2) yielding slightly better results, suggesting that minimizing noise from negative labels improves robustness.
• Parameter sensitivity analysis shows that top-k neighbor selection (k) has an optimal range: too few or too many neighbors degrade performance, indicating a trade-off between information richness and noise.