[Paper Review] ChebNet: Efficient and Stable Constructions of Deep Neural Networks with Rectified Power Units using Chebyshev Approximations.

Shanshan Tang, Bo Li+1 more|arXiv (Cornell University)|Nov 7, 2019

Neural Networks and Applications48 references22 citations

TL;DR

This paper proposes ChebNet, a stable and efficient deep neural network architecture using Chebyshev polynomial approximations to construct rectified power unit (RePU) networks. By replacing power series approximations with hierarchical Chebyshev expansions in the frequency domain, ChebNets achieve the same theoretical approximation bounds as power-series-based RePU networks but with significantly improved numerical stability and performance in practice.

ABSTRACT

In a recent paper [B. Li, S. Tang and H. Yu, arXiv:1903.05858], it was shown that deep neural networks built with rectified power units (RePU) can give better approximation for sufficient smooth functions than those with rectified linear units, by converting polynomial approximation given in power series into deep neural networks with optimal complexity and no approximation error. However, in practice, power series are not easy to compute. In this paper, we propose a new and more stable way to construct deep RePU neural networks based on Chebyshev polynomial approximations. By using a hierarchical structure of Chebyshev polynomial approximation in frequency domain, we build efficient and stable deep neural network constructions. In theory, ChebNets and the deep RePU nets based on Power series have the same upper error bounds for general function approximations. But numerically, ChebNets are much more stable. Numerical results show that the constructed ChebNets can be further trained and obtain much better results than those obtained by training deep RePU nets constructed basing on power series.

Motivation & Objective

• To address the numerical instability and computational difficulty of power series approximations in constructing deep RePU neural networks.
• To develop a more stable alternative to power-series-based RePU networks that maintains optimal approximation complexity.
• To enable practical deployment of deep RePU networks by improving numerical conditioning during construction.
• To demonstrate that Chebyshev-based constructions outperform power-series-based RePU networks in both stability and final model performance after training.

Proposed method

• Constructing deep RePU networks using hierarchical Chebyshev polynomial approximations in the frequency domain to replace traditional power series expansions.
• Leveraging the equioscillation and stability properties of Chebyshev polynomials to improve numerical conditioning during network construction.
• Mapping polynomial approximations onto deep residual-like architectures to achieve optimal approximation complexity.
• Using a frequency-domain decomposition strategy to systematically build multi-level approximations with controlled error bounds.
• Designing a network architecture that mirrors the structure of Chebyshev expansions, enabling efficient and stable parameterization.
• Preserving the same theoretical upper error bounds as power-series-based RePU networks while enhancing numerical robustness.

Experimental results

Research questions

• RQ1 Can Chebyshev polynomial approximations provide a more numerically stable alternative to power series for constructing deep RePU networks?
• RQ2 Does the use of hierarchical Chebyshev approximations maintain the same theoretical approximation error bounds as power-series-based RePU networks?
• RQ3 How does the numerical stability of ChebNets compare to power-series-based RePU networks during training?
• RQ4 Can ChebNets achieve better generalization performance than power-series-based RePU networks after end-to-end training?
• RQ5 What is the impact of frequency-domain decomposition via Chebyshev polynomials on the overall efficiency and convergence of deep RePU networks?

Key findings

• ChebNets achieve the same theoretical upper error bounds for function approximation as power-series-based RePU networks.
• ChebNets demonstrate significantly improved numerical stability compared to power-series-based RePU networks during construction.
• Numerical experiments show that ChebNets can be successfully trained and yield better performance than power-series-based RePU networks after training.
• The hierarchical structure of Chebyshev approximations enables efficient and stable deep network construction with controlled approximation error.
• Chebyshev-based RePU networks outperform their power-series counterparts in practical training scenarios due to enhanced numerical conditioning.
• The proposed method maintains optimal approximation complexity while resolving the instability issues inherent in power series computation.