[Paper Review] Combining Label Propagation and Simple Models Out-performs Graph Neural Networks

Qian Huang, Horace He+3 more|arXiv (Cornell University)|Oct 27, 2020

Advanced Graph Neural Networks48 references114 citations

TL;DR

The paper introduces Correct and Smooth (C&S): a simple, GNN-free pipeline that combines a feature-based base predictor with two post-processing label-propagation steps (error correction and prediction smoothing) to achieve state-of-the-art or near state-of-the-art transductive node classification with far fewer parameters and faster training than GNNs.

ABSTRACT

Graph Neural Networks (GNNs) are the predominant technique for learning over graphs. However, there is relatively little understanding of why GNNs are successful in practice and whether they are necessary for good performance. Here, we show that for many standard transductive node classification benchmarks, we can exceed or match the performance of state-of-the-art GNNs by combining shallow models that ignore the graph structure with two simple post-processing steps that exploit correlation in the label structure: (i) an "error correlation" that spreads residual errors in training data to correct errors in test data and (ii) a "prediction correlation" that smooths the predictions on the test data. We call this overall procedure Correct and Smooth (C&S), and the post-processing steps are implemented via simple modifications to standard label propagation techniques from early graph-based semi-supervised learning methods. Our approach exceeds or nearly matches the performance of state-of-the-art GNNs on a wide variety of benchmarks, with just a small fraction of the parameters and orders of magnitude faster runtime. For instance, we exceed the best known GNN performance on the OGB-Products dataset with 137 times fewer parameters and greater than 100 times less training time. The performance of our methods highlights how directly incorporating label information into the learning algorithm (as was done in traditional techniques) yields easy and substantial performance gains. We can also incorporate our techniques into big GNN models, providing modest gains. Our code for the OGB results is at https://github.com/Chillee/CorrectAndSmooth.

Motivation & Objective

• Motivate how simple models plus label-propagation post-processing can match or exceed GNNs on transductive node classification benchmarks.
• Show that C&S achieves strong performance with dramatically fewer parameters and faster training times.
• Demonstrate that incorporating ground-truth labels into the post-processing steps yields substantial gains.
• Explore how feature augmentation and combining C&S with larger GNNs can provide further improvements.
• Discuss implications for the role of labels and simple post-processing in graph learning.

Proposed method

• Train a base predictor f on node features ignoring the graph (linear model or shallow MLP) to produce base predictions Z.
• Compute residual errors on training nodes E = Z_Lt − Y_Lt and propagate these errors over the graph to obtain a corrected score  hatE via a label-spreading-like diffusion (E^(t+1) = (1-α)E + α S E^(t)).
• Combine the base predictions with the smoothed residuals to get corrected predictions Z^(r) = Z + hatE (with optional autoscale or scaled diffusion variants).
• Perform a second label-propagation step to smooth the final predictions over the graph, using the corrected scores as inputs and resetting training labels to their true values (G^(0) = [Y_Lt on labeled nodes; Z^(r) on unlabeled nodes]); iterate G^(t+1) = (1-α)G^(t) + α S G^(t).
• Optionally augment node features with a spectral embedding before the base prediction to improve performance.
• Show that using ground-truth labels in training/validation can further boost results; compare with and without end-to-end training of GNNs.

Experimental results

Research questions

• RQ1 Can simple, feature-based predictors coupled with graph-based post-processing match or exceed state-of-the-art GNN performance on transductive node classification benchmarks?
• RQ2 How do error-correcting propagation and prediction smoothing contribute to performance gains, and what are the roles of autoscale and fixed-diffusion variants?
• RQ3 Does incorporating ground-truth labels in post-processing (training and validation) substantially improve results compared with using only training labels?
• RQ4 How does C&S compare in training efficiency and parameter count to contemporary GNN models across diverse datasets?
• RQ5 Can C&S be combined with more complex GNNs to yield additional gains?

Key findings

• C&S often exceeds or matches state-of-the-art GNNs on multiple benchmarks using far fewer parameters and with orders of magnitude faster training times.
• On the OGB-Products dataset, C&S surpasses the best known GNN with 137x fewer parameters and >100x less training time.
• Using ground-truth validation labels in the post-processing (Table 4) yields further improvements, with the best model outperforming SOTA on seven of nine datasets.
• Even the Basic Model (base predictor plus label propagation smoothing without labels) can outperform plain GCNs on several datasets, highlighting the value of smoothed outputs.
• C&S with simple base predictors (linear or shallow MLP) and post-processing can outperform or closely rival SOTA baselines on datasets such as Arxiv, Products, Cora, Citeseer, Pubmed, Email, Rice31, US County, and wikiCS; further gains are achieved when combining with feature augmentation and using additional labels.
• The approach demonstrates significant speedups and parameter efficiency, illustrating that directly incorporating label information through classical methods can yield substantial performance gains.