[Paper Review] Dirichlet Energy Constrained Learning for Deep Graph Neural Networks

Kaixiong Zhou, Xiao Huang+5 more|arXiv (Cornell University)|Jul 6, 2021

Advanced Graph Neural NetworksComputer Science51 references39 citations

TL;DR

The paper introduces Dirichlet energy constrained learning to enable deep GNNs by regulating Dirichlet energy per layer, and presents Energetic Graph Neural Networks (EGNN) with orthogonal weight control, lower-bounded residuals, and Shifted ReLU to achieve state-of-the-art results with very deep graphs.

ABSTRACT

Graph neural networks (GNNs) integrate deep architectures and topological structure modeling in an effective way. However, the performance of existing GNNs would decrease significantly when they stack many layers, because of the over-smoothing issue. Node embeddings tend to converge to similar vectors when GNNs keep recursively aggregating the representations of neighbors. To enable deep GNNs, several methods have been explored recently. But they are developed from either techniques in convolutional neural networks or heuristic strategies. There is no generalizable and theoretical principle to guide the design of deep GNNs. To this end, we analyze the bottleneck of deep GNNs by leveraging the Dirichlet energy of node embeddings, and propose a generalizable principle to guide the training of deep GNNs. Based on it, a novel deep GNN framework -- EGNN is designed. It could provide lower and upper constraints in terms of Dirichlet energy at each layer to avoid over-smoothing. Experimental results demonstrate that EGNN achieves state-of-the-art performance by using deep layers.

Motivation & Objective

• Motivate and formalize a generalizable principle to guide the design and training of deep GNNs using Dirichlet energy constraints.
• Develop a deep GNN architecture (EGNN) that satisfies per-layer Dirichlet energy bounds to mitigate over-smoothing and over-separating.
• Show that Dirichlet energy constrained learning can explain and unify existing deep-GNN techniques.
• Empirically demonstrate that EGNN supports very deep architectures (up to 64 layers) with competitive performance on benchmark datasets.

Proposed method

• Define Dirichlet energy E(X^(k)) for node embeddings and establish lower/upper bounds across layers to avoid over-smoothing and over-separating.
• Propose Dirichlet energy constrained learning: c_min E(X^(k-1)) <= E(X^(k)) <= c_max E(X^(0)) with trainable feature transform X^(0)=f(X).
• Design EGNN with three components: (i) orthogonal initialization and regularization of weight matrices to control singular values and satisfy the upper energy bound, (ii) lower-bounded residual connections to maintain a minimum energy per layer, and (iii) Shifted ReLU (SReLU) activation to balance non-linearity and linearity while preserving energy bounds.
• Provide theoretical results (lemmas and propositions) ensuring the orthogonal control, residuals, and activation keep energies within bounds across layers.
• Compare Energy-constrained design to existing normalization, DropEdge, and residual techniques to show generalizability of the principle.

Experimental results

Research questions

• RQ1 Is there a theoretical and generalizable principle to guide architecture design and training of deep GNNs?
• RQ2 How can one design a deep GNN that achieves state-of-the-art performance by stacking many layers while avoiding over-smoothing?
• RQ3 Can a principled energy constraint framework explain and unify existing deep-GNN techniques?
• RQ4 Does EGNN enable stable training and strong performance across benchmark datasets with deep architectures?

Key findings

• EGNN achieves state-of-the-art or competitive node classification performance, especially with deep layers (up to 64) on benchmarks.
• Dirichlet energy at each layer remains within learned lower and upper bounds, validating the constrained learning principle.
• Orthogonal weight initialization and regularization are crucial for maintaining energy bounds and enabling deep training.
• Lower-bounded residual connections and SReLU activation help preserve energy and prevent over-smoothing without sacrificing learning capacity.
• EGNN outperforms several baselines across Cora, Pubmed, Coauthor-Physics, and Ogbn-arxiv, particularly in deep configurations.
• Ablation studies show the importance of each component (orthogonal initialization, lower-bound residuals, and SReLU) for performance and energy stability.