[Paper Review] Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality

Taiji Suzuki|arXiv (Cornell University)|Oct 18, 2018

Image and Signal Denoising Methods81 citations

TL;DR

The paper analyzes deep ReLU networks for functions in Besov and mixed smooth Besov spaces, showing minimax-optimal approximation and estimation rates and that adaptivity helps avoid the curse of dimensionality in mixed smooth spaces.

ABSTRACT

Deep learning has shown high performances in various types of tasks from visual recognition to natural language processing, which indicates superior flexibility and adaptivity of deep learning. To understand this phenomenon theoretically, we develop a new approximation and estimation error analysis of deep learning with the ReLU activation for functions in a Besov space and its variant with mixed smoothness. The Besov space is a considerably general function space including the Holder space and Sobolev space, and especially can capture spatial inhomogeneity of smoothness. Through the analysis in the Besov space, it is shown that deep learning can achieve the minimax optimal rate and outperform any non-adaptive (linear) estimator such as kernel ridge regression, which shows that deep learning has higher adaptivity to the spatial inhomogeneity of the target function than other estimators such as linear ones. In addition to this, it is shown that deep learning can avoid the curse of dimensionality if the target function is in a mixed smooth Besov space. We also show that the dependency of the convergence rate on the dimensionality is tight due to its minimax optimality. These results support high adaptivity of deep learning and its superior ability as a feature extractor.

Motivation & Objective

• Demonstrate that deep ReLU networks can approximate Besov and mixed Besov spaces minimax-optimally.
• Show that deep learning outperforms linear estimators like kernel ridge regression for Besov spaces.
• Establish that mixed smooth Besov spaces allow avoiding the curse of dimensionality with deep nets.
• Provide explicit approximation and estimation error bounds under Besov/mixed Besov assumptions.

Proposed method

• Develop approximation error bounds for ReLU networks approximating Besov and mixed Besov spaces via cardinal B-splines and B-spline representations.
• Prove existence of ReLU networks that approximate B-splines to within epsilon with specified L-∞ error.
• Translate Besov/m-Besov approximation bounds into generalization/estimation error bounds in a nonparametric regression setting.
• Derive minimax optimal rates for estimation in Besov spaces and show improved rates in mixed Besov spaces.
• Compare adaptive deep learning rates to linear (e.g., kernel ridge) rates and establish optimality arguments.

Experimental results

Research questions

• RQ1 Can ReLU-based deep networks achieve minimax-optimal approximation rates for functions in Besov spaces?
• RQ2 Do ReLU networks outperform linear estimators (like kernel ridge regression) for Besov spaces in both approximation and estimation errors?
• RQ3 Does the mixed smooth Besov space allow deep networks to avoid the curse of dimensionality, and what are the resulting rates?
• RQ4 How do the network architecture parameters (depth, width, sparsity, norm bounds) translate into concrete approximation and estimation error bounds?

Key findings

• Deep ReLU networks achieve minimax-optimal approximation rate on Besov spaces under specified smoothness and integrability conditions.
• Deep nets outperform linear estimators such as kernel ridge regression for Besov spaces, especially when the target has spatially inhomogeneous smoothness.
• For mixed smooth Besov spaces, deep networks can avoid the curse of dimensionality and attain near minimax rates, with rates dependent on s and d.
• The approximation error bounds via B-splines yield finite network constructions with explicit L^r norms, showing adaptivity advantages.
• Estimation error analysis shows that under standard nonparametric regression with Gaussian noise, deep nets can achieve the minimax rate n^{-2s/(2s+d)} up to a poly-log factor, which is unattainable by linear estimators.