[Paper Review] Improving Graph Neural Networks with Learnable Propagation Operators

Moshe Eliasof, Lars Ruthotto+1 more|arXiv (Cornell University)|Oct 31, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes ω GNN, a learnable propagation operator framework that enhances Graph Neural Networks (GNNs) by introducing channel-wise trainable weights ω to mix smoothing and sharpening propagation operators. By enabling mixed-sign operators and per-channel learning, ω GNN prevents over-smoothing and improves expressiveness, achieving state-of-the-art performance on 15 real-world datasets in both node- and graph-classification tasks.

ABSTRACT

Graph Neural Networks (GNNs) are limited in their propagation operators. In many cases, these operators often contain non-negative elements only and are shared across channels, limiting the expressiveness of GNNs. Moreover, some GNNs suffer from over-smoothing, limiting their depth. On the other hand, Convolutional Neural Networks (CNNs) can learn diverse propagation filters, and phenomena like over-smoothing are typically not apparent in CNNs. In this paper, we bridge these gaps by incorporating trainable channel-wise weighting factors $ω$ to learn and mix multiple smoothing and sharpening propagation operators at each layer. Our generic method is called $ω$GNN, and is easy to implement. We study two variants: $ω$GCN and $ω$GAT. For $ω$GCN, we theoretically analyse its behaviour and the impact of $ω$ on the obtained node features. Our experiments confirm these findings, demonstrating and explaining how both variants do not over-smooth. Additionally, we experiment with 15 real-world datasets on node- and graph-classification tasks, where our $ω$GCN and $ω$GAT perform on par with state-of-the-art methods.

Motivation & Objective

• Address the limitations of standard GNNs, which use fixed, non-negative propagation operators that lead to over-smoothing and reduced expressiveness.
• Bridge the gap between GNNs and CNNs by enabling learnable, diverse propagation filters similar to depth-wise convolutions in CNNs.
• Overcome the over-smoothing problem in deep GNNs by allowing the network to learn and mix both smoothing and sharpening operators through trainable weights ω.
• Enhance GNN expressiveness by introducing learnable, channel-wise weighting factors that allow flexible mixing of propagation operators.
• Demonstrate the effectiveness of ω GNN across diverse real-world datasets using two variants: ω GCN and ω GAT.

Proposed method

• Introduce a learnable weighting factor ω to control the contribution of multiple propagation operators at each layer, enabling dynamic mixing of smoothing and sharpening effects.
• Formulate ω GNN as a generalization of the standard GNN layer: f^{(l+1)} = σ(S^{(l)} f^{(l)} K^{(l)}), where S^{(l)} is replaced by a learnable, channel-wise weighted combination of base operators.
• Implement two variants: ω GCN using a symmetric normalized adjacency matrix as base operator, and ω GAT using multi-head attention as the base operator.
• Apply a 1×1 convolution with learnable weights ω^{(l)} per channel and layer to mix the contributions of different propagation operators, emulating depth-wise separable convolutions.
• Train the network end-to-end with standard backpropagation, allowing ω to adaptively learn optimal operator mixtures for each layer and channel.
• Theoretical analysis shows that ω scales as 1/L with depth, preventing over-smoothing by maintaining feature diversity across layers.

Experimental results

Research questions

• RQ1 Can learnable, channel-wise weighting factors ω prevent over-smoothing in deep GNNs by enabling mixed-sign propagation operators?
• RQ2 To what extent does the introduction of learnable ω improve the expressiveness of GNNs compared to fixed, non-negative propagation operators?
• RQ3 How does the performance of ω GNN compare to state-of-the-art GNNs across diverse node- and graph-classification benchmarks?
• RQ4 Does the scaling behavior of ω (e.g., ω ∝ 1/L) align with theoretical predictions and contribute to stable training in deep architectures?
• RQ5 Can ω GNN generalize across different GNN architectures, such as GCN and GAT, while maintaining performance and robustness?

Key findings

• ω GNN variants (ω GCN and ω GAT) achieve state-of-the-art performance on 15 real-world datasets, including SOTA results on TUDatasets with 95.2% accuracy on PTC and 94.6% on MUTAG.
• The ablation study confirms that ω GNN prevents over-smoothing: accuracy remains stable or improves with depth, unlike standard GNNs that degrade with more layers.
• The average ω value scales as 1/L across layers, validating the theoretical prediction that ω decreases with depth to maintain feature diversity and avoid over-smoothing.
• ω GNN achieves 85.9% accuracy on Cora with 64 layers (ω GCN), demonstrating robustness to depth, while standard GNNs typically over-smooth beyond 8–16 layers.
• The learned ω values enable the network to mix both smoothing and sharpening operators, as shown in visualization and theoretical analysis, enhancing feature representation beyond what non-negative operators can achieve.
• ω GAT outperforms ω GCN on most benchmarks, achieving 84.4% accuracy on NCI1 and 83.6% on NCI109, indicating the effectiveness of attention-based operator mixing.