[Paper Review] LanczosNet: Multi-Scale Deep Graph Convolutional Networks

Renjie Liao, Zhizhen Zhao+2 more|arXiv (Cornell University)|Jan 6, 2019

Advanced Graph Neural Networks41 references140 citations

TL;DR

LanczosNet constructs low-rank approximations of the graph Laplacian via the Lanczos algorithm to enable efficient multi-scale graph convolutions with learnable spectral filters; AdaLanczosNet back-propagates through Lanczos for graph kernel/node embeddings.

ABSTRACT

We propose the Lanczos network (LanczosNet), which uses the Lanczos algorithm to construct low rank approximations of the graph Laplacian for graph convolution. Relying on the tridiagonal decomposition of the Lanczos algorithm, we not only efficiently exploit multi-scale information via fast approximated computation of matrix power but also design learnable spectral filters. Being fully differentiable, LanczosNet facilitates both graph kernel learning as well as learning node embeddings. We show the connection between our LanczosNet and graph based manifold learning methods, especially the diffusion maps. We benchmark our model against several recent deep graph networks on citation networks and QM8 quantum chemistry dataset. Experimental results show that our model achieves the state-of-the-art performance in most tasks. Code is released at: \url{https://github.com/lrjconan/LanczosNetwork}.

Motivation & Objective

• Motivate effective multi-scale information extraction in graph convolutional networks.
• Provide a scalable method to compute multi-scale graph diffusion without expensive high-power computations.
• Introduce learnable spectral filters within a Lanczos-based framework.
• Offer a variant that back-propagates through the Lanczos process for learning graph kernels or node embeddings.

Proposed method

• Use Lanczos algorithm to obtain a low-rank approximation S ≈ Q T Q^T of the affinity matrix S.
• Construct spectral filters by learning functions of Ritz values/vectors (r_i, v_i) from the Lanczos decomposition via MLPs, enabling learnable spectral filtering.
• Form multi-scale graph convolutions by combining short-scale (via powers of S) and long-scale (via learned spectral filters) components in a differentiable network.
• Optionally, AdaLanczosNet back-propagates through the Lanczos steps to learn graph kernels or node embeddings.
• Relate LanczosNet to diffusion maps by interpreting spectral filtering as operating on diffusion-map-based frequency representations.

Experimental results

Research questions

• RQ1 Can Lanczos-based low-rank Laplacian approximations efficiently capture multi-scale graph information?
• RQ2 Are learnable spectral filters on the Lanczos approximation able to improve performance over fixed spectral filters?
• RQ3 Does back-propagating through the Lanczos process (AdaLanczosNet) yield tangible gains by learning graph kernels or node embeddings?
• RQ4 How does LanczosNet compare to state-of-the-art graph networks on standard benchmarks like citation networks and QM8?

Key findings

• LanczosNet and AdaLanczosNet achieve state-of-the-art performance on several tasks versus 9 recent graph networks.
• The Lanczos-based low-rank approximation enables efficient computation of multi-scale information through Ritz-value-based spectral filtering.
• Learnable spectral filters (via MLPs on Ritz values) increase model capacity over fixed polynomial filters.
• Back-propagating through the Lanczos steps (AdaLanczosNet) provides a differentiable pathway for learning graph kernels and/or node embeddings.