[Paper Review] Analysis of Deep Neural Networks with Quasi-optimal polynomial approximation rates

Joseph Daws, Clayton Webster|arXiv (Cornell University)|Dec 4, 2019

Neural Networks and Applications15 references4 citations

TL;DR

This paper constructs a deep neural network (DNN) that achieves quasi-optimal approximation rates for high-dimensional functions using polynomial basis expansions. By leveraging bounds on polynomial coefficients rather than exact values, the network achieves sub-exponential error decay in M (number of terms) with algebraic complexity in M, matching the performance of best M-term polynomial approximations under mild regularity assumptions.

ABSTRACT

We show the existence of a deep neural network capable of approximating a wide class of high-dimensional approximations. The construction of the proposed neural network is based on a quasi-optimal polynomial approximation. We show that this network achieves an error rate that is sub-exponential in the number of polynomial functions, $M$, used in the polynomial approximation. The complexity of the network which achieves this sub-exponential rate is shown to be algebraic in $M$.

Motivation & Objective

• To establish the existence of a DNN that approximates high-dimensional functions with error rates matching quasi-optimal M-term polynomial approximations.
• To develop a constructive method for building DNNs based on polynomial approximation theory, avoiding reliance on exact coefficient values.
• To analyze the network's approximation error and computational complexity in terms of M, the number of polynomial terms.
• To bridge classical approximation theory with deep learning by showing that DNNs can inherit the convergence rates of polynomial expansions.

Proposed method

• The network architecture is built from subnetworks approximating individual polynomial basis functions Ψν, where ν ranges over a quasi-optimal index set ΛM.
• Each subnetwork Ũν approximates a tensor product polynomial Ψν using a constructive approach based on bounded coefficient estimates from Assumption 2.
• The approximation error is bounded via the triangle inequality: ||u - uNN|| ≤ ||u - uΛM|| + ||uΛM - uNN||, with the first term controlled by quasi-optimal polynomial approximation theory.
• Complexity and depth of each subnetwork Ũν are bounded using logarithmic estimates in terms of the coefficient decay function b(ν), leading to overall network complexity scaling as O(M^{2/d + 1}).
• The final network uNN combines M such subnetworks, each with weights cν, and uses a final layer to sum their outputs.
• The construction relies on uniform bounds on coefficients cν via b(ν), enabling error control without requiring exact coefficient computation.

Experimental results

Research questions

• RQ1 Can a DNN be constructed to achieve the same approximation error rate as quasi-optimal M-term polynomial approximations for high-dimensional functions?
• RQ2 What is the computational complexity (depth and size) of such a DNN in terms of M, the number of polynomial terms?
• RQ3 How does the network’s error depend on the decay rate of polynomial coefficients, as quantified by b(ν)?
• RQ4 Can the network be built without requiring exact knowledge of polynomial coefficients, using only bounds?
• RQ5 Does the resulting DNN inherit the sub-exponential convergence rate of the underlying polynomial approximation?

Key findings

• The constructed DNN achieves an approximation error bound that is sub-exponential in M, matching the best possible rate of quasi-optimal M-term polynomial approximations.
• The network’s overall complexity scales as O(M^{2/d + 1}), which is algebraic in M, ensuring practical feasibility despite high dimensionality.
• The depth of the network grows as O(M^{1/d} log M), which is sub-polynomial and manageable for moderate M.
• The method avoids the need for exact coefficient values by relying on bounds on cν via the function b(ν), making it applicable to problems where coefficients are difficult to compute.
• The error bound for the network is derived via the triangle inequality, with the first term controlled by known results on quasi-optimal polynomial approximation and the second term bounded via subnetwork construction.
• The construction is fully explicit: all network parameters (weights and biases) are defined based on the quasi-optimal index set ΛM and coefficient bounds.