[Paper Review] Spectral Networks and Locally Connected Networks on Graphs

Joan Bruna, Wojciech Zaremba+2 more|arXiv (Cornell University)|Dec 21, 2013

Advanced Graph Neural Networks14 references2,711 citations

TL;DR

The paper extends CNN concepts to graphs using spatial and spectral graph constructions, achieving competitive accuracy with far fewer parameters, including an O(1) parameter regime via smooth spectral multipliers.

ABSTRACT

Convolutional Neural Networks are extremely efficient architectures in image and audio recognition tasks, thanks to their ability to exploit the local translational invariance of signal classes over their domain. In this paper we consider possible generalizations of CNNs to signals defined on more general domains without the action of a translation group. In particular, we propose two constructions, one based upon a hierarchical clustering of the domain, and another based on the spectrum of the graph Laplacian. We show through experiments that for low-dimensional graphs it is possible to learn convolutional layers with a number of parameters independent of the input size, resulting in efficient deep architectures.

Motivation & Objective

• Generalize CNNs to signals on graphs lacking grid structure.
• Develop spatial (locally connected) and spectral (Laplacian-based) network constructions.
• Show parameter efficiency (O(n) and O(1) per layer) and practical performance on graph-structured data.
• Demonstrate deep architectures on low-dimensional graphs with scalable forward propagation.

Proposed method

• Spatial construction: replace grid with graph G=(Ω,W); use multiscale clustering to form Ω_k and neighborhoods N_k; layer k computes x_{k+1,j}=L_k h( sum_i F_{k,i,j} x_{k,i} ) with sparse F_{k,i,j} and pooling L_k over clusters; parameter count is O(n).
• Spectral construction: transform signals via graph Laplacian eigenvectors V; layers operate as x_{k+1,j}= h( V sum_i F_{k,i,j} V^T x_{k,i} ); uses diagonal F_{k,i,j} in spectral domain; with d eigenvectors kept, parameters per layer are O(n) and can be O(1) under spectral multiplier smoothness.
• O(1) parameter construction: propose smooth spectral multipliers by ordering eigenvectors by eigenvalues and using spline-based parameterization to achieve constant per-filter parameters; discuss dual graph for spectral coordinates.

Experimental results

Research questions

• RQ1 Can CNN-like convolutional operations be meaningfully defined on arbitrary graphs without translation symmetry?
• RQ2 Do spatially or spectrally inspired graph convolutions provide parameter-efficient deep networks with competitive performance on graph-structured data?
• RQ3 Can we achieve O(1) or near-constant parameter counts per layer while preserving performance on low-dimensional graphs?
• RQ4 How do smooth spectral multipliers influence localization and generalization of learned filters on graphs?
• RQ5 What is the relationship between classical CNNs and graph-based convolution through spectra of the graph Laplacian?

Key findings

• Graph-based CNNs can achieve competitive or superior accuracy with far fewer parameters than fully connected nets on MNIST variants.
• Spatial locally connected networks with multiscale clustering outperform vanilla fully connected models on subsampled MNIST.
• Spectral constructions perform competitively; smoothing spectral multipliers improves localization and generalization of filters.
• An O(1) parameter regime is feasible using smooth spectral multipliers, indicating potential scalability to large graphs.
• On MNIST-sphere with rotations, graph-based architectures significantly outperform nearest neighbor and often match or exceed fully connected baselines with far fewer parameters.