[Paper Review] Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't

E Weinan, Chao Ma+2 more|arXiv (Cornell University)|Sep 22, 2020

Neural Networks and Applications85 references50 citations

TL;DR

The paper surveys the current mathematical understanding of neural network-based learning, focusing on approximation, generalization, loss landscapes, and training dynamics, with emphasis on over-parameterization and implicit regularization.

ABSTRACT

The purpose of this article is to review the achievements made in the last few years towards the understanding of the reasons behind the success and subtleties of neural network-based machine learning. In the tradition of good old applied mathematics, we will not only give attention to rigorous mathematical results, but also the insight we have gained from careful numerical experiments as well as the analysis of simplified models. Along the way, we also list the open problems which we believe to be the most important topics for further study. This is not a complete overview over this quickly moving field, but we hope to provide a perspective which may be helpful especially to new researchers in the area.

Motivation & Objective

• Explain the reasons behind the success and fragility of neural network-based learning.
• Identify and formalize the function spaces and norms that govern approximation and generalization.
• Discuss the loss landscape, optimization dynamics, and implicit regularization in training.
• Outline key results from the numerical-analysis perspective and highlight major open questions.

Proposed method

• Review universal approximation results and their quantitative limitations (e.g., Barron-type results) for high-dimensional function approximation.
• Introduce and analyze random feature models and the associated RKHS as a natural hypothesis space.
• Develop two-layer neural network theory through Barron spaces and direct/inverse approximation theorems.
• Discuss residual and multi-layer networks via depth-related function spaces and depth separation concepts.
• Examine loss landscapes using high-dimensional analogies and mean-field/gradient dynamics results.
• Present Rademacher complexity-based generalization bounds and their implications for learnability and estimation error.

Experimental results

Research questions

• RQ1 What are the natural function spaces associated with common neural network architectures (e.g., two-layer networks) that control approximation and generalization?
• RQ2 How do approximation error and estimation error trade off in high-dimensional, often over-parameterized settings?
• RQ3 What roles do loss landscapes and training dynamics play in selecting solutions with good generalization?
• RQ4 Can implicit regularization from optimization dynamics replace explicit regularization in achieving robust generalization?
• RQ5 What are the limitations and open problems in connecting numerical analysis intuitions to practical deep learning models?

Key findings

• For random feature models, the direct approximation error decays as 1/m with the Barron norm controlling the rate.
• Two-layer networks can approximate Barron functions with an L2 error of order 1/sqrt(m) and an L∞ error with rate depending on dimensionality.
• Barron space provides a natural function space for two-layer networks; Barron norms bound both approximation and generalization aspects.
• Rademacher complexity bounds imply generalization gaps of order ||f*||_*^2/m + ||f*||_* / sqrt(n) in ideal settings, illustrating the trade-off between model size and data.
• In over-parameterized regimes, global minima exist and training dynamics (implicit regularization) can influence which minima are selected, affecting generalization.
• Depth-related analyses (e.g., residual networks) and mean-field scaling provide qualitative insights into training dynamics and convergence, with many open questions remaining.