[Paper Review] Interpreting and Unifying Graph Neural Networks with An Optimization Framework

Meiqi Zhu, Xiao Wang+3 more|arXiv (Cornell University)|Jan 28, 2021

Advanced Graph Neural Networks49 references28 citations

TL;DR

The paper presents a unified optimization framework that explains various GNN propagation mechanisms as solutions to a feature fitting plus graph Laplacian regularization objective, and derives new GNNs with low-pass and high-pass filters within this framework.

ABSTRACT

Graph Neural Networks (GNNs) have received considerable attention on graph-structured data learning for a wide variety of tasks. The well-designed propagation mechanism which has been demonstrated effective is the most fundamental part of GNNs. Although most of GNNs basically follow a message passing manner, litter effort has been made to discover and analyze their essential relations. In this paper, we establish a surprising connection between different propagation mechanisms with a unified optimization problem, showing that despite the proliferation of various GNNs, in fact, their proposed propagation mechanisms are the optimal solution optimizing a feature fitting function over a wide class of graph kernels with a graph regularization term. Our proposed unified optimization framework, summarizing the commonalities between several of the most representative GNNs, not only provides a macroscopic view on surveying the relations between different GNNs, but also further opens up new opportunities for flexibly designing new GNNs. With the proposed framework, we discover that existing works usually utilize naive graph convolutional kernels for feature fitting function, and we further develop two novel objective functions considering adjustable graph kernels showing low-pass or high-pass filtering capabilities respectively. Moreover, we provide the convergence proofs and expressive power comparisons for the proposed models. Extensive experiments on benchmark datasets clearly show that the proposed GNNs not only outperform the state-of-the-art methods but also have good ability to alleviate over-smoothing, and further verify the feasibility for designing GNNs with our unified optimization framework.

Motivation & Objective

• Motivate a macroscopic, principled view of diverse GNN propagation mechanisms.
• Propose a unified optimization objective combining feature fitting and graph regularization.
• Show how representative GNNs (GCN, SGC, PPNP/APPNP, JKNet, DAGNN) fit into the framework.
• Design two novel GNNs with adjustable low-pass and high-pass graph kernels.
• Provide theoretical analysis on convergence and expressive power, plus empirical validation.

Proposed method

• Define a two-term objective combining a feature fitting term and a graph Laplacian regularization term.
• Introduce flexible graph convolutional kernels F1 and F2 to realize different filtering behavior.
• Show that several GNNs correspond to special cases of the objective with specific F1, F2, and regularization parameters.
• Derive closed-form/iterative solutions for propagation under the unified objective.
• Develop two new GNN models (GNN-LF and GNN-HF) with low-pass and high-pass kernels and analyze their convergence and expressiveness.
• Provide theoretical connections (Theorems) linking existing models to the unified framework and discuss implications for design.

Experimental results

Research questions

• RQ1 Can common GNN propagation mechanisms be unified under a single optimization framework with a feature fitting term and a Laplacian regularization term?
• RQ2 How do different choices of graph convolutional kernels (F1, F2) correspond to known GNNs, and can this inspire new architectures?
• RQ3 Do novel low-pass and high-pass kernels within the framework alleviate over-smoothing and improve expressiveness?
• RQ4 What are the convergence properties and expressive power of the proposed GNN-LF and GNN-HF models?
• RQ5 Can the framework guide principled design of future GNNs with interpretable objectives?

Key findings

• A unified objective is able to reproduce propagation mechanisms of several representative GNNs, linking them through a two-term optimization.
• GNNs like SGC/GCN correspond to a purely graph regularization objective, while APPNP/PPNP introduce a fitting term that encodes original features.
• The framework enables designing two novel GNNs with adjustable low-pass and high-pass filtering kernels that can alleviate over-smoothing.
• Theoretical analyses provide convergence and expressive power comparisons for the proposed models.
• Extensive experiments show the proposed GNNs outperform state-of-the-art methods and verify the feasibility of designing GNNs under the unified framework.