[Paper Review] Rational Neural Networks for Approximating Jump Discontinuities of Graph Convolution Operator

Zhiqian Chen, Feng Chen+3 more|arXiv (Cornell University)|Aug 30, 2018

Advanced Graph Neural Networks51 references4 citations

TL;DR

This paper proposes RationalNet, a novel graph neural network that leverages rational function approximation to effectively model jump discontinuities in graph signals, avoiding costly eigenvector matrix multiplications through a reformulation in terms of the graph Laplacian. The method achieves exponential convergence on jump signals and outperforms polynomial-based models and baselines in both synthetic and real-world graph datasets with significantly lower error and competitive runtime.

ABSTRACT

For node level graph encoding, a recent important state-of-art method is the graph convolutional networks (GCN), which nicely integrate local vertex features and graph topology in the spectral domain. However, current studies suffer from several drawbacks: (1) graph CNNs relies on Chebyshev polynomial approximation which results in oscillatory approximation at jump discontinuities; (2) Increasing the order of Chebyshev polynomial can reduce the oscillations issue, but also incurs unaffordable computational cost; (3) Chebyshev polynomials require degree $Ω$(poly(1/$ε$)) to approximate a jump signal such as $|x|$, while rational function only needs $\mathcal{O}$(poly log(1/$ε$))\cite{liang2016deep,telgarsky2017neural}. However, it's non-trivial to apply rational approximation without increasing computational complexity due to the denominator. In this paper, the superiority of rational approximation is exploited for graph signal recovering. RatioanlNet is proposed to integrate rational function and neural networks. We show that rational function of eigenvalues can be rewritten as a function of graph Laplacian, which can avoid multiplication by the eigenvector matrix. Focusing on the analysis of approximation on graph convolution operation, a graph signal regression task is formulated. Under graph signal regression task, its time complexity can be significantly reduced by graph Fourier transform. To overcome the local minimum problem of neural networks model, a relaxed Remez algorithm is utilized to initialize the weight parameters. Convergence rate of RatioanlNet and polynomial based methods on jump signal is analyzed for a theoretical guarantee. The extensive experimental results demonstrated that our approach could effectively characterize the jump discontinuities, outperforming competing methods by a substantial margin on both synthetic and real-world graphs.

Motivation & Objective

• Address the limitation of polynomial-based graph convolutional networks in approximating jump discontinuities due to Gibbs phenomenon and slow convergence.
• Overcome the computational burden of rational approximation in spectral graph methods by avoiding eigenvector matrix multiplication.
• Develop a neural network architecture that integrates rational functions with graph convolution for improved signal recovery on non-smooth graph signals.
• Provide theoretical justification for the faster convergence of rational functions compared to polynomials on jump signals.
• Improve training stability and convergence by using a relaxed Remez algorithm for weight initialization.

Proposed method

• Reformulate rational functions of eigenvalues as functions of the graph Laplacian to eliminate the need for eigenvector matrix multiplication.
• Formulate graph signal regression as a learning task and leverage graph Fourier transform to reduce time complexity.
• Introduce a relaxed Remez algorithm to initialize network weights, mitigating local minima issues in training.
• Use a neural network to learn the coefficients of rational functions, enabling end-to-end training on graph-structured data.
• Theoretical analysis shows that rational approximation achieves O(poly log(1/ε)) degree for ε-approximation, while polynomials require Ω(poly(1/ε)).
• The model is trained using a loss function minimizing mean squared error between predicted and target signals on graph-embedded data.

Experimental results

Research questions

• RQ1 Can rational function approximation outperform polynomial approximation in modeling jump discontinuities in graph signals?
• RQ2 Can rational functions be efficiently integrated into spectral graph neural networks without incurring high computational costs from eigenvector operations?
• RQ3 Does the proposed RationalNet model achieve faster convergence on jump signals compared to polynomial-based GCNs?
• RQ4 Can a relaxed Remez algorithm effectively initialize neural network weights to improve convergence and avoid local minima?
• RQ5 How does RationalNet perform in practice on real-world and synthetic graph datasets with non-smooth signals?

Key findings

• RationalNet achieved a mean squared error (MSE) of 3e-5 on the Minnesota road network, significantly outperforming the next best baseline (PolyFit at 0.0008).
• On the Fairfax road network, RationalNet maintained the lowest MSE (3e-5), while SVR(RBF) and PolyNet achieved 0.0055 and 0.0039 respectively.
• RationalNet converged within 300 iterations, taking less than one minute on both Minnesota (2642 nodes) and Fairfax (993 nodes) datasets, despite higher per-iteration cost than baselines.
• Theoretical analysis confirmed that rational approximation converges exponentially faster than polynomial approximation on jump signals, requiring only O(poly log(1/ε)) degrees compared to Ω(poly(1/ε)) for polynomials.
• Visualization of eigenvectors confirmed that high-frequency components (e.g., φ₇₀₁, φ₇₀₂) capture non-smooth, discontinuous signals, which RationalNet effectively models.
• Dirichlet energy analysis showed that the reconstructed signal ζ using top 50% high-frequency components had a very high energy (15,384.10), confirming its non-smooth nature, which RationalNet accurately captured.