[Paper Review] Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusion

Haotian Ju, Dongyue Li+2 more|arXiv (Cornell University)|Feb 9, 2023

Advanced Graph Neural Networks4 citations

TL;DR

This paper improves generalization bounds for graph neural networks (GNNs) by replacing the reliance on maximum degree with the spectral norm of the graph diffusion matrix, yielding significantly tighter bounds. Using PAC-Bayesian analysis and Hessian-based stability measures, the authors derive upper and matching lower bounds that scale with the spectral norm, validated empirically on real-world graphs where these bounds are orders of magnitude smaller than prior methods.

ABSTRACT

Graph neural networks are widely used tools for graph prediction tasks. Motivated by their empirical performance, prior works have developed generalization bounds for graph neural networks, which scale with graph structures in terms of the maximum degree. In this paper, we present generalization bounds that instead scale with the largest singular value of the graph neural network's feature diffusion matrix. These bounds are numerically much smaller than prior bounds for real-world graphs. We also construct a lower bound of the generalization gap that matches our upper bound asymptotically. To achieve these results, we analyze a unified model that includes prior works' settings (i.e., convolutional and message-passing networks) and new settings (i.e., graph isomorphism networks). Our key idea is to measure the stability of graph neural networks against noise perturbations using Hessians. Empirically, we find that Hessian-based measurements correlate with the observed generalization gaps of graph neural networks accurately. Optimizing noise stability properties for fine-tuning pretrained graph neural networks also improves test performance on several graph-level classification tasks.

Motivation & Objective

• To address the weakness of existing generalization bounds for GNNs that scale with maximum graph degree, which often leads to vacuous bounds.
• To develop tighter, non-vacuous generalization bounds that reflect real-world graph structures more accurately.
• To unify analysis across GNN architectures, including message-passing, GCN, and GIN models, under a single theoretical framework.
• To establish a theoretical link between Hessian-based stability and observed generalization gaps in GNNs.
• To enable improved fine-tuning of pretrained GNNs by optimizing for noise stability.

Proposed method

• Propose a unified model framework that includes GCNs, message-passing networks, and GINs for joint theoretical analysis.
• Apply PAC-Bayesian generalization theory to bound the generalization gap via stability against noise perturbations.
• Use Hessian matrices of the loss with respect to model weights to quantify stability and derive bounds.
• Replace the maximum degree dependence in prior bounds with the spectral norm of the graph diffusion matrix $P_G^{l-1}$.
• Derive an upper bound on the generalization gap that scales with $\|P_G^{l-1}\|$, the spectral norm of the diffusion matrix.
• Construct a matching lower bound instance showing the generalization gap asymptotically scales with $\|P_G^{l-1}\|$.

(a) Spectral norm vs. max degree for five graphs

Experimental results

Research questions

• RQ1 Can generalization bounds for GNNs be improved by replacing maximum degree with spectral norm of the diffusion matrix?
• RQ2 How does Hessian-based stability correlate with actual generalization gaps in GNNs?
• RQ3 Can the proposed bounds be unified across different GNN architectures like GCN, MPNN, and GIN?
• RQ4 Is the spectral norm a better complexity measure than maximum degree for quantifying GNN generalization?
• RQ5 Can optimizing noise stability during fine-tuning improve test performance on graph-level tasks?

Key findings

• The proposed generalization bounds scale with the spectral norm of the graph diffusion matrix $P_G^{l-1}$, which is orders of magnitude smaller than the maximum degree on real-world graphs.
• Empirical results show that Hessian-based stability measurements correlate strongly with observed generalization gaps across multiple GNN models and datasets.
• The upper bound on the generalization gap matches the lower bound asymptotically, confirming the tightness of the spectral norm as a complexity measure.
• On five real-world graph datasets, the spectral norm-based bounds are significantly tighter than prior bounds that scale with maximum degree.
• Optimizing for noise stability during fine-tuning improves test performance on graph-level classification tasks, demonstrating practical utility.
• The theoretical framework unifies analysis across GCN, MPNN, and GIN models, showing consistent dependence on $\|P_G^{l-1}\|$.

(b) Comparing generalization bounds