[Paper Review] Implicit Graph Neural Networks

Fangda Gu, Heng Chang+3 more|arXiv (Cornell University)|Sep 14, 2020

Advanced Graph Neural NetworksComputer Science53 references45 citations

TL;DR

IGNN introduces an implicit fixed-point graph neural network framework with PF-based well-posedness, trained via projected gradient and implicit differentiation, achieving strong long-range dependency modeling and extending to heterogeneous graphs.

ABSTRACT

Graph Neural Networks (GNNs) are widely used deep learning models that learn meaningful representations from graph-structured data. Due to the finite nature of the underlying recurrent structure, current GNN methods may struggle to capture long-range dependencies in underlying graphs. To overcome this difficulty, we propose a graph learning framework, called Implicit Graph Neural Networks (IGNN), where predictions are based on the solution of a fixed-point equilibrium equation involving implicitly defined "state" vectors. We use the Perron-Frobenius theory to derive sufficient conditions that ensure well-posedness of the framework. Leveraging implicit differentiation, we derive a tractable projected gradient descent method to train the framework. Experiments on a comprehensive range of tasks show that IGNNs consistently capture long-range dependencies and outperform the state-of-the-art GNN models.

Motivation & Objective

• Address the limitations of finite-hop GNNs in capturing long-range dependencies on graphs.
• Provide a mathematical well-posedness framework for equilibrium GNNs using Perron-Frobenius theory.
• Develop a tractable training method via projected gradient descent and implicit differentiation.
• Extend IGNN to heterogeneous networks with relation-specific parameters.
• Demonstrate empirical superiority of IGNN over state-of-the-art GNNs across node and graph classification tasks.

Proposed method

• Define the IGNN prediction as Y = f_Θ(X) with X solving the equilibrium X = φ(W X A + b_Ω(U)).
• Use a CONE (component-wise non-expansive) activation φ and PF theory to derive sufficient conditions for existence and uniqueness of X.
• Train with projected gradient descent enforcing a well-posedness constraint ||W||_∞ ≤ κ/λ_pf(A) via projection, and compute exact gradients through implicit differentiation.
• Extend to heterogeneous networks: X = φ(∑_i (W_i X A_i + b_{Ω_i}(U_i))).
• Provide numerically tractable convex conditions for training in heterogeneous settings: ∑_i ||A_i||_1 ||W_i||_∞ ≤ κ < 1.
• Show that for positively homogeneous φ, PF conditions can be satisfied by rescaling W (Theorem 4.3).

Experimental results

Research questions

• RQ1 Can equilibrium (implicit) GNNs be guaranteed to have a unique solution for node representations under graph and activation choices?
• RQ2 Can IGNN be trained efficiently with exact gradients using implicit differentiation while maintaining well-posedness?
• RQ3 Do IGNNs better capture long-range dependencies than finite-iteration GNNs across diverse tasks?
• RQ4 How does IGNN perform on heterogeneous graphs with multiple relation types compared to state-of-the-art baselines?
• RQ5 What empirical gains do IGNN provide for node and graph classification benchmarks?

Key findings

• IGNN consistently captures long-range dependencies and outperforms state-of-the-art GNNs on a wide range of tasks.
• On synthetic Chains data, IGNN outperforms finite-iteration GNNs as chain length grows, while finite T-iteration GNNs fail with longer chains.
• On PPI node classification, IGNN achieves a Micro-F1 of 97.6%, higher than SSE (83.6%) and slightly above GAT (97.3%).
• On the Amazon multi-label node classification task, IGNN outperforms baselines as supervision increases, with robust long-range dependency handling.
• Graph classification results show IGNN achieving the best performance in 4 of 5 datasets, with per-dataset bests including MUTAG: 89.3±6.7, PTC: 70.1±5.6, COX2: 86.9±4.0, PROTEINS: 77.7±3.4, NCI1: 80.5±1.9.
• On heterogeneous networks ( ACM, IMDB, DBLP ), IGNN attains top performance on ACM and IMDB and competitive results on DBLP (best on ACM/IMDB; competitive on DBLP).