[Paper Review] Bag of Tricks for Node Classification with Graph Neural Networks

Yangkun Wang, Jiarui Jin+4 more|arXiv (Cornell University)|Mar 24, 2021

Advanced Graph Neural NetworksComputer Science35 references28 citations

TL;DR

The paper surveys practical tricks for node classification with GNNs, introduces new label usage, robust loss, and architecture tweaks, and shows consistent empirical gains across multiple datasets.

ABSTRACT

Over the past few years, graph neural networks (GNN) and label propagation-based methods have made significant progress in addressing node classification tasks on graphs. However, in addition to their reliance on elaborate architectures and algorithms, there are several key technical details that are frequently overlooked, and yet nonetheless can play a vital role in achieving satisfactory performance. In this paper, we first summarize a series of existing tricks-of-the-trade, and then propose several new ones related to label usage, loss function formulation, and model design that can significantly improve various GNN architectures. We empirically evaluate their impact on final node classification accuracy by conducting ablation studies and demonstrate consistently-improved performance, often to an extent that outweighs the gains from more dramatic changes in the underlying GNN architecture. Notably, many of the top-ranked models on the Open Graph Benchmark (OGB) leaderboard and KDDCUP 2021 Large-Scale Challenge MAG240M-LSC benefit from these techniques we initiated.

Motivation & Objective

• Motivate the importance of data processing and training tricks in improving node classification accuracy beyond architectural changes.
• Introduce novel techniques for using labels as input and for label reuse to enhance semi-supervised learning.
• Propose a robust loss function family to improve classification under outliers and noisy labels.
• Explore tweaks to GAT architectures and loss design to boost performance across diverse graphs.

Proposed method

• Propose label usage as input where ground-truth labels are provided during training and inference to propagate label information.
• Develop label reuse to feed predicted labels from previous iterations as inputs during training.
• Introduce a robust loss function family, including Loge loss, for classification with improved robustness to outliers.
• Tweak GNN architectures, such as GCN and GAT, including symmetric normalization and edge-feature aware variants.
• Evaluate proposed tricks across multiple datasets (Cora, Citeseer, Pubmed, Reddit, ogbn-arxiv, ogbn-proteins, ogbn-products) and compare variants using ablation studies.

Experimental results

Research questions

• RQ1 Can explicit use of training labels as input during both training and inference improve node classification performance without causing degenerate solutions?
• RQ2 Does feeding predicted labels from previous iterations (label reuse) improve semi-supervised GNN performance across diverse graphs?
• RQ3 How do robust, non-convex-like loss functions (e.g., Loge loss) compare to traditional losses in GNN training for node classification?
• RQ4 What GAT and adjacency normalization variants yield the best performance across different graph datasets?
• RQ5 Do the proposed tricks collectively account for the performance gains observed in top-performing OGB leaderboard models?

Key findings

• Label-as-input and label-reuse techniques consistently improve accuracy on large datasets with moderate label rates.
• The Loge loss often outperforms logistic and Savage losses across multiple datasets, with favorable gradient properties near the decision boundary.
• GAT variants with symmetric normalized adjacency typically outperform vanilla GAT on several datasets, and edge-feature variants can yield substantial gains on protein data.
• Combining label-usage tricks with existing methods (e.g., C&S) yields further improvements.
• On large-scale datasets like ogbn-proteins, the proposed GAT with edge features achieved notable ROC-AUC improvements, indicating strong robustness and applicability.