[Paper Review] Deep neural network approximation theory for high-dimensional functions

Pierfrancesco Beneventano, Patrick Cheridito+3 more|arXiv (Cornell University)|Dec 29, 2021

Neural Networks and Applications4 citations

TL;DR

This paper establishes that deep neural networks (DNNs) can approximate a broad class of high-dimensional functions without suffering from the curse of dimensionality. By introducing approximation spaces and proving closure properties for compositions of locally Lipschitz functions, maxima, and products, the authors show that DNNs require only polynomial growth in parameters—relative to input dimension and inverse accuracy—enabling efficient approximation of complex, high-dimensional functions.

ABSTRACT

The purpose of this article is to develop machinery to study the capacity of deep neural networks (DNNs) to approximate high-dimensional functions. In particular, we show that DNNs have the expressive power to overcome the curse of dimensionality in the approximation of a large class of functions. More precisely, we prove that these functions can be approximated by DNNs on compact sets such that the number of parameters necessary to represent the approximating DNNs grows at most polynomially in the reciprocal $1/\varepsilon$ of the approximation accuracy $\varepsilon>0$ and in the input dimension $d\in \mathbb{N} =\{1,2,3,\dots\}$. To this end, we introduce certain approximation spaces, consisting of sequences of functions that can be efficiently approximated by DNNs. We then establish closure properties which we combine with known and new bounds on the number of parameters necessary to approximate locally Lipschitz continuous functions, maximum functions, and product functions by DNNs. The main result of this article demonstrates that DNNs have sufficient expressiveness to approximate certain sequences of functions which can be constructed by means of a finite number of compositions using locally Lipschitz continuous functions, maxima, and products without the curse of dimensionality.

Motivation & Objective

• To develop a theoretical framework for analyzing the expressive power of deep neural networks (DNNs) in approximating high-dimensional functions.
• To investigate whether DNNs can approximate high-dimensional functions with parameter counts growing polynomially in dimension and inverse accuracy, avoiding the curse of dimensionality.
• To establish closure properties for approximation spaces under composition with locally Lipschitz functions, maxima, and products.
• To prove that DNNs can efficiently approximate functions constructed through finite compositions of basic operations and regular functions without exponential parameter growth.

Proposed method

• Introduce approximation spaces consisting of sequences of functions that can be efficiently approximated by DNNs.
• Define and analyze key operations—local Lipschitz continuity, maximum functions, and product functions—within the DNN approximation framework.
• Use linear interpolation and parameter-efficient DNN constructions to approximate locally Lipschitz continuous functions with polynomial parameter growth.
• Construct explicit DNN representations for maxima and products using activation functions and network architectures that preserve approximation efficiency.
• Prove closure of the approximation space under composition with locally Lipschitz functions, maxima, and products using network parallelization and composition rules.
• Leverage network clipping and matrix projection operations (e.g., via $\mathbf{A}_{P_d,0}$) to ensure output dimensionality and approximation accuracy control.

Experimental results

Research questions

• RQ1 Can deep neural networks approximate high-dimensional functions with parameter counts growing polynomially in dimension and inverse accuracy?
• RQ2 Do DNNs maintain polynomial tractability when approximating compositions of locally Lipschitz functions, maxima, and products?
• RQ3 What structural properties of DNNs enable them to avoid the curse of dimensionality in function approximation?
• RQ4 How do closure properties of approximation spaces under composition affect the expressiveness of DNNs?
• RQ5 Can explicit DNN constructions for maxima and products be integrated into larger networks while preserving polynomial parameter growth?

Key findings

• DNNs can approximate a large class of high-dimensional functions without the curse of dimensionality, as the number of parameters grows at most polynomially in the input dimension $d$ and the inverse accuracy $\varepsilon^{-1}$.
• The number of parameters required to approximate a function within accuracy $\varepsilon$ on compact sets is bounded by $\mathcal{P}(\mathscr{F}_{d,\varepsilon}) \leq c d^c \varepsilon^{-c}$ for some constant $c \in \mathbb{R}$, demonstrating polynomial tractability.
• Approximation spaces are closed under composition with locally Lipschitz continuous functions, maximum functions, and product functions, enabling hierarchical construction of complex functions.
• Explicit DNN representations for maxima and products are constructed with controlled parameter counts, ensuring that these operations do not introduce exponential complexity.
• The composition of DNNs approximating locally Lipschitz functions, maxima, and products results in a DNN whose realization approximates the target function with error $\leq \varepsilon$ on $[-r,r]^d$.
• The use of projection matrices $P_d$ and network clipping operations ensures that the final DNN output matches the desired scalar output function $p_d(x) = x_d$ with bounded error.