[Paper Review] Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spaces

Philipp Grohs, Felix Voigtlaender|arXiv (Cornell University)|Apr 6, 2021

Stochastic Gradient Optimization Techniques5 citations

TL;DR

This paper proves the existence of a theory-to-practice gap in deep learning by establishing sampling complexity lower bounds for neural network approximation spaces. It demonstrates that while neural networks can theoretically approximate functions with high rates, practical algorithms like stochastic gradient descent cannot achieve these rates due to inherent algorithmic limitations, confirming a long-observed empirical discrepancy.

ABSTRACT

We study the computational complexity of (deterministic or randomized) algorithms based on point samples for approximating or integrating functions that can be well approximated by neural networks. Such algorithms (most prominently stochastic gradient descent and its variants) are used extensively in the field of deep learning. One of the most important problems in this field concerns the question of whether it is possible to realize theoretically provable neural network approximation rates by such algorithms. We answer this question in the negative by proving hardness results for the problems of approximation and integration on a novel class of neural network approximation spaces. In particular, our results confirm a conjectured and empirically observed theory-to-practice gap in deep learning. We complement our hardness results by showing that approximation rates of a comparable order of convergence are (at least theoretically) achievable.

Motivation & Objective

• To investigate whether theoretically provable neural network approximation rates can be realized by practical algorithms based on point samples.
• To formally establish the existence of a theory-to-practice gap in deep learning, where theoretical approximation rates are not achievable by standard optimization algorithms.
• To introduce a novel class of neural network approximation spaces that encode constraints on network depth, weight magnitude, and approximation error decay.
• To derive sampling complexity lower bounds for approximation and integration tasks on these spaces, showing fundamental algorithmic limitations.
• To reconcile the gap between the superior theoretical expressivity of neural networks and the practical inefficacy of optimization algorithms in achieving theoretical rates.

Proposed method

• The authors define neural network approximation spaces $ A^{eta,p}_{oldsymbol{ u},oldsymbol{c}}([0,1]^d) $ that characterize functions based on how rapidly their approximation error decays with increasing network size, incorporating constraints on depth $ oldsymbol{ u} $, weight magnitude $ oldsymbol{c} $, and $ L^p $-norm error.
• They analyze the sampling complexity of deterministic and randomized algorithms that use point samples to approximate or integrate functions in these spaces.
• Using variational and minimax techniques, they derive lower bounds on the number of samples required to achieve a given approximation accuracy, showing that the required sample size grows faster than what is achievable by standard stochastic optimization.
• The analysis involves constructing specific test functions and using duality arguments to bound the error of any algorithm in terms of its sample size and the function's smoothness and network complexity.
• Key inequalities and optimization over parameters $ heta, heta_0, heta^* $ are used to derive tight bounds on the best possible convergence rates, leading to the main hardness results.
• The proof relies on constructing a family of functions in the unit ball of the approximation space and showing that no algorithm can achieve the theoretical approximation rate without an infeasible number of samples.

Experimental results

Research questions

• RQ1 Can the theoretically optimal approximation rates of deep neural networks be achieved by practical algorithms based on point samples?
• RQ2 What is the fundamental sampling complexity required to approximate or integrate functions in neural network approximation spaces?
• RQ3 Why do common deep learning optimization algorithms fail to achieve the theoretical approximation rates predicted by function space analysis?
• RQ4 To what extent do constraints on network depth and weight size limit the feasibility of achieving theoretical approximation rates in practice?
• RQ5 Is there a provable gap between the theoretical approximation power of neural networks and the practical performance of optimization algorithms?

Key findings

• The paper proves that there exists a provable theory-to-practice gap in deep learning: even though neural networks can theoretically approximate certain functions with high rates, no deterministic or randomized algorithm using point samples can achieve these rates.
• Sampling complexity lower bounds are derived for approximation and integration on neural network approximation spaces, showing that the number of samples required grows faster than what is feasible with standard stochastic optimization.
• For functions in the unit ball of $ A^{eta,p}_{oldsymbol{ u},oldsymbol{c}}([0,1]^d) $, the best achievable convergence rate is bounded below by $ n^{-eta} $, but this rate cannot be realized by any algorithm using $ n $ samples.
• The hardness results are established via a novel analysis of the interplay between smoothness $ eta $, network depth $ oldsymbol{ u} $, and weight constraints $ oldsymbol{c} $, showing that these constraints fundamentally limit algorithmic performance.
• The paper shows that while approximation rates of comparable order are theoretically achievable, they are not accessible via standard deep learning algorithms, confirming the empirical observation of the theory-to-practice gap.
• The results are formalized through tight bounds on the infimum of maxima of functions involving parameters $ heta, heta_0, heta^* $, leading to explicit expressions for the minimal achievable convergence rate under given constraints.