[Paper Review] Unifying over-smoothing and over-squashing in graph neural networks: A physics informed approach and beyond

Zhiqi Shao, Dai Shi+4 more|arXiv (Cornell University)|Sep 6, 2023

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes a physics-informed graph neural network, MHKG and its generalization G-MHKG, which unifies the mitigation of over-smoothing and over-squashing by reversing the time direction of the graph heat equation to generate high-pass filtering effects. The method uses multi-scale heat kernel filtering functions to balance smoothing and sharpening of node features, revealing a fundamental trade-off between over-smoothing and over-squashing, with G-MHKG achieving state-of-the-art performance on both homophilic and heterophilic graphs.

ABSTRACT

Graph Neural Networks (GNNs) have emerged as one of the leading approaches for machine learning on graph-structured data. Despite their great success, critical computational challenges such as over-smoothing, over-squashing, and limited expressive power continue to impact the performance of GNNs. In this study, inspired from the time-reversal principle commonly utilized in classical and quantum physics, we reverse the time direction of the graph heat equation. The resulted reversing process yields a class of high pass filtering functions that enhance the sharpness of graph node features. Leveraging this concept, we introduce the Multi-Scaled Heat Kernel based GNN (MHKG) by amalgamating diverse filtering functions' effects on node features. To explore more flexible filtering conditions, we further generalize MHKG into a model termed G-MHKG and thoroughly show the roles of each element in controlling over-smoothing, over-squashing and expressive power. Notably, we illustrate that all aforementioned issues can be characterized and analyzed via the properties of the filtering functions, and uncover a trade-off between over-smoothing and over-squashing: enhancing node feature sharpness will make model suffer more from over-squashing, and vice versa. Furthermore, we manipulate the time again to show how G-MHKG can handle both two issues under mild conditions. Our conclusive experiments highlight the effectiveness of proposed models. It surpasses several GNN baseline models in performance across graph datasets characterized by both homophily and heterophily.

Motivation & Objective

• Address the long-standing challenge of unifying the treatment of over-smoothing, over-squashing, and limited expressive power in GNNs.
• Investigate the underlying relationship between over-smoothing and over-squashing by drawing inspiration from time-reversal in physics.
• Develop a flexible, analyzable GNN framework that allows systematic control over feature smoothing, topological compression, and expressive power.
• Demonstrate that over-smoothing and over-squashing are fundamentally linked through filtering function properties in the spectral domain.

Proposed method

• Reverse the time direction of the graph heat equation to generate a time-reversed process that acts as a high-pass filter, enhancing feature sharpness.
• Construct the Multi-Scaled Heat Kernel GNN (MHKG) by combining low-pass (smoothing) and high-pass (sharpening) filtering functions derived from heat and reverse heat kernels.
• Generalize MHKG into G-MHKG to allow flexible filtering functions and control over model components, including learnable parameters and spectral filtering matrices.
• Use spectral decomposition of the graph Laplacian to express filtering operations in the eigenbasis, enabling analytical control over energy dynamics and feature propagation.
• Introduce a time-manipulation strategy in G-MHKG to adaptively handle both over-smoothing and over-squashing, especially effective for heterophily graphs.
• Leverage the tightness principle of graph framelets to interpret G-MHKG-F as a non-tight variant, enabling robust performance on diverse graph types.

Figure 1 : Top: The evolution of the node feature of diffusion (smoothing) process (i.e., from distinct features to over-smoothing). Bottom: the reverse diffusion (sharpening) process (i.e., from nearly identical to distinct node features).

Experimental results

Research questions

• RQ1 Can over-smoothing and over-squashing in GNNs be unified under a single theoretical framework using principles from physics?
• RQ2 What is the intrinsic relationship between over-smoothing and over-squashing, and can it be characterized via filtering function properties?
• RQ3 Can a single GNN architecture effectively mitigate both over-smoothing and over-squashing simultaneously, and under what conditions?
• RQ4 How do different filtering functions (e.g., sinusoidal, exponential) affect model performance across homophilic and heterophilic graphs?
• RQ5 To what extent does the tightness of framelet-based filtering matter in practical graph learning tasks?

Key findings

• The proposed G-MHKG model achieves state-of-the-art performance on both homophilic (Cora, Citeseer, PubMed) and heterophilic (Cornell, Texas, Wisconsin) graph benchmarks.
• G-MHKG-F achieves 88.2% accuracy on Cornell, 86.1% on Texas, and 84.7% on Wisconsin, outperforming baseline models like GAT, GIN, and UFG.
• A fundamental trade-off between over-smoothing and over-squashing is revealed: enhancing feature sharpness (via high-pass filtering) increases over-squashing, and vice versa.
• Time manipulation in G-MHKG enables effective handling of both issues for heterophily graphs, but it is theoretically impossible to fully resolve both simultaneously in homophily graphs.
• The MHKG-I variant underperforms due to non-positive definite filtering matrices caused by negative eigenvalue scaling, highlighting the importance of proper filtering function design.
• G-MHKG-F performs comparably or better than the tight graph framelet (UFG) on heterophily graphs, suggesting that tightness may not be essential for practical performance.

Figure 2 : The figure on the left represents different types of HFD filtering outcomes and the trade-off between two issues. One can check that to induce more sharpening (filtering function from bottom to top), the model will suffer more from OSQ. The figure on the right illustrates the situation de