[Paper Review] New error bounds for deep networks using sparse grids

Hadrien Montanelli, Qiang Du|arXiv (Cornell University)|Dec 23, 2017

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper establishes new error bounds for deep ReLU neural networks approximating multivariate functions in Korobov spaces by leveraging sparse grid approximations. It demonstrates that the curse of dimensionality is significantly lessened, achieving approximation complexity with depth and size scaling as $\mathcal{O}(|\log_2 \epsilon| \log_2 d)$ and $\mathcal{O}(\epsilon^{-1/2} |\log_2 \epsilon|^{3(d-1)/2 + 1} \log_2 d)$, respectively, by exploiting a binary tree structure to emulate sparse grid refinement.

ABSTRACT

We prove a theorem concerning the approximation of multivariate functions by deep ReLU networks. We present new error estimates for which the curse of the dimensionality is lessened by establishing a connection with sparse grids.

Motivation & Objective

• To address the curse of dimensionality in deep learning approximation by connecting deep ReLU networks to sparse grid theory.
• To establish rigorous upper bounds on approximation error for functions in Korobov spaces using deep networks.
• To demonstrate that deep networks can achieve better convergence rates than shallow networks by exploiting hierarchical sparse grid structures.
• To show that the network architecture is independent of the target function, with only weights adapting to the function.

Proposed method

• The method constructs a deep ReLU network to approximate sparse grid interpolants of functions in Korobov spaces using a hierarchical basis representation.
• It employs a binary tree structure to emulate the refinement process of sparse grids, enabling efficient depth and width control.
• The network uses basis functions $\phi_{\boldsymbol{l},\boldsymbol{i}}$ with coefficients $v_{\boldsymbol{l},\boldsymbol{i}}$ that decay as $2^{-|\boldsymbol{l}|_1}$, ensuring stability and convergence.
• The approximation error is bounded by splitting it into two parts: the truncation error of the sparse grid expansion and the network approximation error of the basis functions.
• The network depth is controlled by the logarithmic scaling of the refinement level $\ell$, with depth $\mathcal{O}(|\log_2 \delta| \log_2 d)$, where $\delta$ controls the truncation error.
• The size of the network is derived from the number of active nodes in the binary tree, scaling with the number of basis functions and the refinement level.

Experimental results

Research questions

• RQ1 Can deep ReLU networks achieve better approximation rates than shallow networks for high-dimensional functions by leveraging sparse grid structures?
• RQ2 To what extent does the use of sparse grids reduce the curse of dimensionality in deep network approximation?
• RQ3 How can the hierarchical structure of sparse grids be emulated by deep ReLU networks using a binary tree architecture?
• RQ4 What is the dependence of network depth and size on the desired accuracy $\epsilon$ and dimension $d$ in this framework?
• RQ5 Can the same network architecture approximate all functions in a Korobov space, with only weights adjusted per function?

Key findings

• The approximation error is bounded by $\|f - f_N\|_{\infty} \leq \epsilon$ using a network with depth $\mathcal{O}(|\log_2 \epsilon| \log_2 d)$.
• The network size scales as $\mathcal{O}(\epsilon^{-1/2} |\log_2 \epsilon|^{3(d-1)/2 + 1} \log_2 d)$, showing a significant reduction in the exponential dependence on dimension compared to classical bounds.
• The method achieves a lessened curse of dimensionality by exploiting the anisotropic structure of sparse grids, which aligns well with axis-aligned function behavior.
• The network architecture is independent of the target function, with only the weights $v_{\boldsymbol{l},\boldsymbol{i}}$ adjusted, enabling a universal design for a function class.
• The proof relies on controlling coefficient decay and basis function approximation via a binary tree, ensuring that the total error remains bounded by $\epsilon$ when $\delta = \epsilon/2$.
• The approach generalizes to any expansion with coefficients satisfying $\ell^2$-decay and efficiently implementable basis functions, suggesting broad applicability.