[Paper Review] Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionality

Ryumei Nakada, Masaaki Imaizumi|arXiv (Cornell University)|Jul 4, 2019

Neural Networks and Applications43 references44 citations

TL;DR

The paper proves that DNN approximation and generalization rates depend on the intrinsic Minkowski dimension of the data, not the nominal ambient dimension, and shows minimax optimality and broader applicability.

ABSTRACT

In this study, we prove that an intrinsic low dimensionality of covariates is the main factor that determines the performance of deep neural networks (DNNs). DNNs generally provide outstanding empirical performance. Hence, numerous studies have actively investigated the theoretical properties of DNNs to understand their underlying mechanisms. In particular, the behavior of DNNs in terms of high-dimensional data is one of the most critical questions. However, this issue has not been sufficiently investigated from the aspect of covariates, although high-dimensional data have practically low intrinsic dimensionality. In this study, we derive bounds for an approximation error and a generalization error regarding DNNs with intrinsically low dimensional covariates. We apply the notion of the Minkowski dimension and develop a novel proof technique. Consequently, we show that convergence rates of the errors by DNNs do not depend on the nominal high dimensionality of data, but on its lower intrinsic dimension. We further prove that the rate is optimal in the minimax sense. We identify an advantage of DNNs by showing that DNNs can handle a broader class of intrinsic low dimensional data than other adaptive estimators. Finally, we conduct a numerical simulation to validate the theoretical results.

Motivation & Objective

• Motivate and formalize the role of intrinsic dimensionality in DNN performance for high-dimensional data.
• Define Minkowski dimension and relate it to intrinsic data structure.
• Derive approximation and generalization bounds for DNNs that depend on intrinsic dimension d rather than ambient dimension D.
• Show minimax optimality of the derived rates.
• Provide numerical simulations validating the theoretical results.

Proposed method

• Model nonparametric regression with f0 in Hölder class on [0,1]^D and covariate distribution with Minkowski-dimension d.
• Use ReLU-based DNN realizations with controlled width, depth, and parameter scale to approximate f0.
• Partition the domain into hypercubes to construct local Taylor-based trapezoid-type approximators, then combine via a max(x) aggregation to control error accumulation.
• Derive approximation rate: ||R(Ψ)−f0||_{L∞(μ)} = O(W^{−β/d}) with W parameter count.
• Establish generalization bounds using empirical process theory and local Rademacher complexity to obtain n-dependent rates: ||f̂ − f0||_{L2(μ)}^2 ≤ C n^{−2β/(2β+d)} up to log factors.
• Provide minimax lower bound showing the rate is near-optimal: inf f̂ sup (… ) ≥ C′ n^{−2β/(2β+d)}.

Experimental results

Research questions

• RQ1 Does the intrinsic Minkowski dimension of the data support determine the convergence rates of DNN approximation and generalization?
• RQ2 Can DNNs achieve faster rates than traditional high-dimensional bounds when data lie on low-dimensional (possibly non-smooth) sets?
• RQ3 Is the rate dependent on the intrinsic dimension d minimax optimal for Hölder-smooth targets?
• RQ4 Do DNNs offer advantages over kernel/Gaussian process estimators under general intrinsic dimensional structures?
• RQ5 Do finite-depth DNNs suffice to attain the proposed rates?

Key findings

• Approximation error scales as W^{−β/d}, with d the Minkowski intrinsic dimension, not the ambient D (dimension).
• Generalization error scales as n^{−2β/(2β+d)} (up to logarithmic factors), independent of the ambient dimension D.
• Rates are shown to be minimax optimal up to logarithmic factors via a matching lower bound.
• DNNs can handle a broader class of intrinsic low-dimensional data than some adaptive estimators, including non-smooth fractal-like supports.
• Numerical simulations corroborate the theoretical rates and illustrate the effect of smaller intrinsic dimension on performance.