[Paper Review] On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topology

Francesco Di Giovanni, Lorenzo Giusti+4 more|arXiv (Cornell University)|Feb 6, 2023

Advanced Memory and Neural Computing31 citations

TL;DR

The paper provides a theoretical analysis of over-squashing in MPNNs, showing that width can mitigate the issue at a potential generalization cost, depth generally does not help and can cause vanishing gradients, and graph topology (especially commute time) is the dominant factor influencing over-squashing.

ABSTRACT

Message Passing Neural Networks (MPNNs) are instances of Graph Neural Networks that leverage the graph to send messages over the edges. This inductive bias leads to a phenomenon known as over-squashing, where a node feature is insensitive to information contained at distant nodes. Despite recent methods introduced to mitigate this issue, an understanding of the causes for over-squashing and of possible solutions are lacking. In this theoretical work, we prove that: (i) Neural network width can mitigate over-squashing, but at the cost of making the whole network more sensitive; (ii) Conversely, depth cannot help mitigate over-squashing: increasing the number of layers leads to over-squashing being dominated by vanishing gradients; (iii) The graph topology plays the greatest role, since over-squashing occurs between nodes at high commute (access) time. Our analysis provides a unified framework to study different recent methods introduced to cope with over-squashing and serves as a justification for a class of methods that fall under graph rewiring.

Motivation & Objective

• Understand how width affects over-squashing and its trade-offs in generalization.
• Investigate whether increasing depth mitigates over-squashing and identify regimes where it worsens training.
• Analyze how graph topology and spectrum influence over-squashing beyond local curvature bounds.
• Provide a unified framework relating spatial and spectral rewiring methods to over-squashing.
• Offer analytical justification for graph rewiring approaches as mitigations for over-squashing.

Proposed method

• Extend the sensitivity analysis to high-dimensional node features in MPNNs with a width parameter p.
• Study MPNN dynamics via Jacobian bounds: ||∂h_v^(m)/∂h_u^(0)||_L1 ≤ (c_sigma p w)^m (S_r,a^m)_{vu}.
• Show that width can mitigate over-squashing when the topology-induced term decays with distance.
• Analyze depth regimes: shallow-diameter regime where over-squashing arises among distant nodes and deep regime where vanishing gradients dominate.
• Relate over-squashing to graph topology through access time, commute time, and Cheeger-related spectral properties.
• Derive conditions linking Jacobian sensitivity and graph walks to topology-driven over-squashing (Theorem 4.1), and gravity of vanishing gradients under spectral norm constraints (Theorem 4.2).
• Provide a framework connecting structural rewiring (spatial and spectral) to reductions in over-squashing (via topological/spectral measures).

Experimental results

Research questions

• RQ1 Does increasing width mitigate over-squashing, and what is the cost for generalization?
• RQ2 Can deeper networks alleviate over-squashing, or do they incur vanishing gradients or other drawbacks?
• RQ3 How does graph topology, including commute time and spectrum, influence over-squashing beyond local curvature arguments?
• RQ4 Do spatial and spectral rewiring strategies consistently mitigate over-squashing, and can they be unified under a single framework?
• RQ5 Under what conditions do Jacobian-based sensitivity measures predict when over-squashing occurs for distant node interactions?

Key findings

• Width mitigates over-squashing by increasing the model capacity and Lipschitz-related factors, but may harm generalization.
• Depth does not generally fix over-squashing; when m is comparable to the graph diameter, over-squashing occurs for distant node interactions, and with large m gradients can vanish.
• Topology, via minimum degree and walk counts, strongly governs the distance at which over-squashing arises, often more than curvature considerations.
• Over-squashing correlates with access time between nodes, and commute time/Effective Resistance framework explains the effectiveness of spatial and spectral rewiring methods.
• In the deep regime, gradients decay exponentially with layer count under certain spectral-norm bounds, illustrating a fundamental limitation of deep MPNNs for long-range dependencies.
• The work provides a unified interpretation of various rewiring approaches (spatial and spectral) as mitigating factors tied to graph topology and spectrum.