[Paper Review] Deep ReLU networks overcome the curse of dimensionality for bandlimited functions

Hadrien Montanelli, Haizhao Yang+1 more|arXiv (Cornell University)|Jan 1, 2019

Neural Networks and Applications22 references35 citations

TL;DR

This paper proves that deep ReLU networks overcome the curse of dimensionality for generalized bandlimited functions by leveraging Maurey's theorem and efficient approximation of Chebyshev polynomials and analytic functions. It establishes that approximation to accuracy $\epsilon$ in the $L^2$-norm requires depth $L = O(\log^2(1/\epsilon))$ and size $W = O(\epsilon^{-2} \log^2(1/\epsilon))$, independent of dimension for bandlimited functions on the unit cube.

ABSTRACT

We prove a theorem concerning the approximation of bandlimited multivariate functions by deep ReLU networks for which the curse of the dimensionality is overcome. Our theorem is based on a result by Maurey and on the ability of deep ReLU networks to approximate Chebyshev polynomials and analytic functions efficiently.

Motivation & Objective

• To address the curse of dimensionality in high-dimensional function approximation using deep neural networks.
• To establish theoretical bounds on the approximation power of deep ReLU networks for generalized bandlimited functions.
• To demonstrate that deep ReLU networks can achieve dimension-independent approximation rates for this class of functions.
• To extend existing results on shallow networks to deep architectures by exploiting compositional structure and efficient multiplication networks.

Proposed method

• Utilizes Maurey's theorem to bound approximation error in terms of entropy numbers of function classes.
• Constructs deep ReLU networks that efficiently approximate multivariate monomials $x_1 \cdots x_d$ using recursive composition of binary multiplication units.
• Employs a network architecture that approximates Chebyshev polynomials and analytic functions with depth and size scaling logarithmically in the inverse accuracy.
• Applies the approximation of monomials to construct networks for general polynomials with bounded coefficients.
• Uses the structure of bandlimited functions expressed as integrals over Fourier transforms to link approximation of kernel functions to network design.
• Derives error bounds in the $L^2(B, \mu)$-norm by combining polynomial approximation with measure-dependent constants.

Experimental results

Research questions

• RQ1 Can deep ReLU networks overcome the curse of dimensionality for bandlimited functions in high dimensions?
• RQ2 What is the minimal depth and size required for a deep ReLU network to approximate bandlimited functions to a given accuracy $\epsilon$?
• RQ3 How does the approximation error scale with respect to $\epsilon$ and dimension $d$ for this function class?
• RQ4 Can deep ReLU networks efficiently approximate the kernel functions $K(w \cdot x)$ that define generalized bandlimited functions?
• RQ5 What role do Chebyshev polynomials and analytic function approximation play in achieving dimension-independent error bounds?

Key findings

• Deep ReLU networks achieve $L^2$-approximation error $\epsilon$ for generalized bandlimited functions with depth $L = O(\log^2(1/\epsilon))$ and size $W = O(\epsilon^{-2} \log^2(1/\epsilon))$, independent of dimension $d$.
• The approximation error for the product $x_1 \cdots x_d$ over $[-M,M]^d$ is bounded by $\epsilon$ using a network of depth and size $O(d \log^2(d/\epsilon) + d^2 \log^2 M)$.
• Polynomials of degree $n$ with bounded coefficients can be approximated to accuracy $\epsilon$ using networks of depth and size $O(n \log^2(Cn/\epsilon) + n^2 \log^2 M)$.
• The approximation of analytic functions and Chebyshev polynomials by deep ReLU networks enables the construction of efficient approximants for bandlimited functions.
• The error bounds are independent of dimension $d$ for the unit cube $B = [0,1]^d$ under a probability measure $\mu$, thus overcoming the curse of dimensionality.
• The theoretical framework shows that very deep networks are more efficient than shallow ones in terms of both approximation and parallel computing performance.