[Paper Review] Estimation of a function of low local dimensionality by deep neural networks

Michael Köhler, Adam Krzyżak+1 more|arXiv (Cornell University)|Aug 29, 2019

Statistical Methods and Inference4 citations

TL;DR

This paper demonstrates that deep neural networks (DNNs) can achieve optimal convergence rates in nonparametric regression by exploiting locally low-dimensional structures in high-dimensional data. By constructing a DNN-based estimator that adapts to the local dimensionality $d^*$, the authors show the rate of convergence is $n^{-2p/(2p+d^*)}$, effectively circumventing the curse of dimensionality when $d^* \ll d$. The method uses piecewise polynomial approximation via B-splines and hierarchical partitioning to model functions with localized low-dimensional dependencies.

ABSTRACT

Deep neural networks (DNNs) achieve impressive results for complicated tasks like object detection on images and speech recognition. Motivated by this practical success, there is now a strong interest in showing good theoretical properties of DNNs. To describe for which tasks DNNs perform well and when they fail, it is a key challenge to understand their performance. The aim of this paper is to contribute to the current statistical theory of DNNs. We apply DNNs on high dimensional data and we show that the least squares regression estimates using DNNs are able to achieve dimensionality reduction in case that the regression function has locally low dimensionality. Consequently, the rate of convergence of the estimate does not depend on its input dimension $d$, but on its local dimension $d^*$ and the DNNs are able to circumvent the curse of dimensionality in case that $d^*$ is much smaller than $d$. In our simulation study we provide numerical experiments to support our theoretical result and we compare our estimate with other conventional nonparametric regression estimates. The performance of our estimates is also validated in experiments with real data.

Motivation & Objective

• To develop a theoretical framework explaining why deep neural networks perform well on high-dimensional data with locally low-dimensional structure.
• To address the challenge of the curse of dimensionality in nonparametric regression by leveraging local dimensionality $d^*$ instead of global input dimension $d$.
• To establish that DNNs can achieve optimal convergence rates $n^{-2p/(2p+d^*)}$ when the regression function has locally low-dimensional structure.
• To provide a theoretical justification for the empirical success of DNNs in tasks like image and speech recognition, where data often exhibit local low-dimensional manifolds.

Proposed method

• The method constructs a DNN-based regression estimator using piecewise polynomial approximation via B-spline basis functions.
• It employs hierarchical partitioning of the input space into regions where the regression function depends on at most $d^*$ variables, exploiting local low-dimensional structure.
• The approximation uses truncated B-spline basis functions $B_{k,j}^{(trunc)}$ to represent indicator functions of local regions, enabling piecewise approximation.
• Each local component $f_k(x_{J_k})$ is approximated using tensor-product B-splines $B_{{\mathbf{j}},M,{\mathbf{t}}}$ on $d^*$-dimensional subspaces.
• The final DNN estimator is a linear combination of these basis functions with coefficients $\gamma_j$, chosen to minimize $L_2$ error.
• The construction ensures the number of parameters scales with $n^{d^*/(2p + d^*)}$, leading to optimal convergence rates.

Experimental results

Research questions

• RQ1 Can deep neural networks achieve optimal convergence rates in nonparametric regression when the underlying function has locally low-dimensional structure rather than global smoothness?
• RQ2 Does the use of locally adaptive basis functions in DNNs allow circumvention of the curse of dimensionality when $d^* \ll d$?
• RQ3 What is the theoretical rate of convergence of DNN-based estimators under local low-dimensional assumptions?
• RQ4 How does the approximation error of the DNN estimator depend on the local dimension $d^*$ and smoothness $p$?
• RQ5 Can the proposed method outperform classical nonparametric estimators in high-dimensional settings with sparse local structure?

Key findings

• The DNN-based regression estimator achieves a convergence rate of $n^{-2p/(2p + d^*)}$, which depends only on the local dimension $d^*$ and not the global input dimension $d$, thus avoiding the curse of dimensionality.
• The rate is optimal up to logarithmic factors, matching the minimax lower bound for $(p,C)$-smooth functions on $d^*$-dimensional manifolds.
• The method constructs a DNN using $J \sim 2^{K_1} \cdot K_2 \cdot n^{d^*/(2p + d^*)}$ basis functions, with coefficients bounded in terms of the sup-norm of local components.
• The approximation error is bounded by $c_{46} \cdot K_2 \cdot n^{-p/(2p + d^*)}$, which decays as $n$ increases, provided $K_1$ and $K_2$ are chosen appropriately.
• The $L_2$ risk of the estimator is bounded by $c_{50} \cdot 2^{K_1} \cdot K_2 \cdot n^{-2p/(2p + d^*)}$, confirming the rate is achievable under the given assumptions.
• Numerical experiments and real data validation confirm that the DNN estimator outperforms conventional nonparametric methods in high-dimensional, locally low-dimensional settings.