[Paper Review] Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels

Simon S. Du, Kangcheng Hou+4 more|arXiv (Cornell University)|May 30, 2019

Advanced Graph Neural NetworksComputer Science33 references74 citations

TL;DR

Introduces Graph Neural Tangent Kernels (GNTKs), a kernel that corresponds to infinitely wide GNNs trained by gradient descent, achieving strong graph classification performance while retaining kernel ease of training and analysis.

ABSTRACT

While graph kernels (GKs) are easy to train and enjoy provable theoretical guarantees, their practical performances are limited by their expressive power, as the kernel function often depends on hand-crafted combinatorial features of graphs. Compared to graph kernels, graph neural networks (GNNs) usually achieve better practical performance, as GNNs use multi-layer architectures and non-linear activation functions to extract high-order information of graphs as features. However, due to the large number of hyper-parameters and the non-convex nature of the training procedure, GNNs are harder to train. Theoretical guarantees of GNNs are also not well-understood. Furthermore, the expressive power of GNNs scales with the number of parameters, and thus it is hard to exploit the full power of GNNs when computing resources are limited. The current paper presents a new class of graph kernels, Graph Neural Tangent Kernels (GNTKs), which correspond to infinitely wide multi-layer GNNs trained by gradient descent. GNTKs enjoy the full expressive power of GNNs and inherit advantages of GKs. Theoretically, we show GNTKs provably learn a class of smooth functions on graphs. Empirically, we test GNTKs on graph classification datasets and show they achieve strong performance.

Motivation & Objective

• Motivate combining the benefits of graph kernels (easy training, theoretical guarantees) with graph neural networks (expressive power).
• Propose a general recipe to convert a wide class of GNNs into Graph Neural Tangent Kernels (GNTKs).
• Provide theoretical guarantees showing GNTKs learn smooth graph functions with polynomial sample complexity under certain conditions.
• Empirically validate GNTKs on standard graph classification benchmarks and compare with state-of-the-art methods.

Proposed method

• Translate a broad class of GNNs (including GIN, GCN, and JK-Net) into corresponding GNTKs by deriving closed-form, layer-wise covariance recursions.
• Compute pairwise kernel values Theta(G, G') analytically without training by leveraging infinite-width NTK theory.
• Use a dynamic programming approach to propagate covariance matrices Sigma and intermediate kernels Theta through BLOCK and READOUT operations.
• Provide a Readout step (with or without Jumping Knowledge) to obtain the final GNTK value for graph pairs.
• Offer a theoretical generalization analysis for a simple GNN-based GNTK, including sample complexity under certain data-generation assumptions.

Experimental results

Research questions

• RQ1 Can we construct a kernel that captures the expressive power of GNNs while retaining the trainability and guarantees of graph kernels?
• RQ2 How to systematically translate common GNN architectures into corresponding GNTKs and compute pairwise graph similarities efficiently?
• RQ3 Do GNTKs provide theoretical generalization guarantees and polynomial-sample learnability for smooth graph functions?
• RQ4 How do GNTKs perform on standard graph classification benchmarks compared to state-of-the-art GNNs and graph kernels?

Key findings

• GNTKs achieve state-of-the-art or strong performance on several graph classification datasets, notably 83.6% on COLLAB and 67.9% on PTC, outperforming many baselines.
• GNTKs often outperform corresponding GNN baselines on social networks and bioinformatics graphs, while also offering faster computation in some settings (e.g., IMDB-B).
• Empirical results indicate GNTK performance improves with deeper aggregation (more BLOCKs) and with Jumping Knowledge in many cases, reflecting inheritance of GNN advantages.
• The paper provides a first polynomial-sample-size generalization analysis for a GK/GNN fusion, showing certain GNTK variants can learn smooth graph functions with polynomial sample complexity.
• GNTKs maintain kernel-level tractability for training and analysis, while capturing the high-order, multi-layer information characteristic of GNNs.