[Paper Review] A New Perspective on Machine Learning: How to do Perfect Supervised Learning

Hui Jiang|arXiv (Cornell University)|Jan 7, 2019

Machine Learning and Algorithms22 references4 citations

TL;DR

This paper proposes a new theoretical framework for supervised machine learning by incorporating bandlimiting—rooted in physical realizability—showing that all practical learning tasks are asymptotically solvable with sufficient data and a sufficiently complex, bandlimited model. The key result is that generalization error converges to zero as training data size increases, resolving the paradox of overfitting avoidance in large neural networks.

ABSTRACT

In this work, we introduce the concept of bandlimiting into the theory of machine learning because all physical processes are bandlimited by nature, including real-world machine learning tasks. After the bandlimiting constraint is taken into account, our theoretical analysis has shown that all practical machine learning tasks are asymptotically solvable in a perfect sense. Furthermore, the key towards this solvability almost solely relies on two factors: i) a sufficiently large amount of training samples beyond a threshold determined by a difficulty measurement of the underlying task; ii) a sufficiently complex and bandlimited model. Moreover, for some special cases, we have derived new error bounds for perfect learning, which can quantify the difficulty of learning. These generalization bounds are not only asymptotically convergent but also irrelevant to model complexity. Our new results on generalization have provided a new perspective to explain the recent successes of large-scale supervised learning using complex models like neural networks.

Motivation & Objective

• To resolve the theoretical-practice gap in machine learning, particularly why large neural networks generalize well despite high capacity.
• To formalize the physical constraint of bandlimiting in real-world learning tasks, acknowledging that all physical processes are inherently bandlimited.
• To establish conditions under which supervised learning becomes asymptotically perfect, independent of model complexity in the limit.
• To derive new generalization error bounds that quantify learning difficulty and are independent of model complexity.
• To explain the success of deep learning through the lens of bandlimited function approximation and sufficient data scaling.

Proposed method

• Introduces a deterministic function-fitting formulation where the target function $ f(oldsymbol{x}) $ is assumed to be bandlimited, reflecting physical realizability.
• Applies multivariate Taylor expansion with orthogonal monomial bases to represent both the target function and the learned model $ \hat{f}(oldsymbol{x}) $, with error terms bounded by $ \xi_n(\boldsymbol{x}) $.
• Uses a multivariate Vandermonde matrix constructed from training samples to solve for model coefficients, proving invertibility with probability one when samples are i.i.d. from $ p(\boldsymbol{x}) $.
• Demonstrates that as $ N \to \infty $, the difference between true and learned coefficients vanishes, and the residual error $ \boldsymbol{\xi}_N \to 0 $, ensuring convergence of $ \hat{f} \to f $.
• Extends the proof to approximately bandlimited functions by introducing a small out-of-band residual $ \epsilon $, maintaining convergence under $ B_\epsilon $-bandlimiting.
• Proves equivalence of any two learned models $ \hat{f}_1 $ and $ \hat{f}_2 $ in the limit $ N \to \infty $, showing $ \|\hat{f}_1 - \hat{f}_2\| \to 0 $ in expectation.

Experimental results

Research questions

• RQ1 Why do large neural networks generalize well in practice despite theoretical predictions of overfitting?
• RQ2 Can supervised learning be asymptotically perfect under realistic physical constraints such as bandlimiting?
• RQ3 What is the minimal data requirement for perfect learning, and how is it related to the intrinsic difficulty of the task?
• RQ4 How can generalization error bounds be derived that are independent of model complexity?
• RQ5 To what extent do bandlimited models and sufficient data ensure convergence to the true function?

Key findings

• All practical machine learning tasks are asymptotically solvable in a perfect sense when the target function and model are bandlimited and sufficient training data is available.
• The expected generalization error $ R(\hat{f}|\mathcal{D}_N) \to 0 $ as $ N \to \infty $, with the error bound decaying as $ \frac{(KB^\prime U)^{n+1}H}{(n+1)!} \to 0 $, independent of model complexity.
• The required number of training samples $ N $ scales as $ O((N^{1/K})) $, where $ K $ is the input dimension, linking data size to model capacity via the degree of the Taylor expansion.
• The multivariate Vandermonde matrix formed from training samples is invertible with probability one when samples are i.i.d., ensuring unique solution recovery in the limit.
• For approximately bandlimited functions, the convergence result holds by replacing the true bandlimit $ B $ with an $ \epsilon $-approximate bandlimit $ B_\epsilon $, maintaining error bounds.
• Any two learned models $ \hat{f}_1 $ and $ \hat{f}_2 $ converge to each other in $ L^2 $-norm under $ p(\boldsymbol{x}) $ as $ N \to \infty $, proving model equivalence in the asymptotic regime.