[Paper Review] Deep ReLU network approximation of functions on a manifold

Johannes Schmidt-Hieber|arXiv (Cornell University)|Aug 2, 2019

Neural Networks and Applications27 references53 citations

TL;DR

The paper shows sparsely connected deep ReLU networks can approximate Hölder functions defined on a d*-dimensional manifold embedded in higher dimensions, achieving error ε with O(ε^{-d*/β} log(1/ε)) non-zero parameters, and derives statistical risk bounds for ERM over such networks.

ABSTRACT

Whereas recovery of the manifold from data is a well-studied topic, approximation rates for functions defined on manifolds are less known. In this work, we study a regression problem with inputs on a $d^*$-dimensional manifold that is embedded into a space with potentially much larger ambient dimension. It is shown that sparsely connected deep ReLU networks can approximate a Hölder function with smoothness index $β$ up to error $ε$ using of the order of $ε^{-d^*/β}\log(1/ε)$ many non-zero network parameters. As an application, we derive statistical convergence rates for the estimator minimizing the empirical risk over all possible choices of bounded network parameters.

Motivation & Objective

• Motivate the study of function approximation when inputs lie on an unknown d*-dimensional manifold embedded in a higher-dimensional space.
• Develop a deep ReLU network construction that exploits manifold structure to achieve favorable approximation rates.
• Establish explicit parameter-count bounds and error rates linking ε, d*, β, and log factors.
• Derive statistical risk bounds for ERM over sparsely connected deep ReLU networks in the manifold regression setting.

Proposed method

• Define deep ReLU networks with bounded weights and sparse connectivity.
• Develop a network construction that leverages a partition of unity and local chart approximations on the manifold.
• Use smooth local coordinates and Taylor-type approximations to build local networks; combine them via network composition rules (including skip/identity via ReLU properties).
• Prove an approximation bound on Hölder functions on compact manifolds with smooth local coordinates that yields the rate ε^{-d*/β} up to log terms.
• Show that all network parameters can be kept bounded by one without sacrificing approximation quality (bounded-parameter regime).
• Translate the approximation error into a statistical risk bound for the empirical risk minimizer over the network class (oracle-type inequality).

Experimental results

Research questions

• RQ1 Can sparsely connected deep ReLU networks approximate Hölder functions defined on a d*-dimensional manifold with rate ε^{-d*/β} (up to logarithmic factors) when inputs lie on the manifold?
• RQ2 How many non-zero network parameters are needed to achieve an ε-approximation on the manifold, and can weights be bounded by one?
• RQ3 What are the statistical risk implications for ERM over such network classes when data lie on an unknown manifold?
• RQ4 How can local coordinate charts and partitions of unity be integrated into a deep network to exploit manifold structure for improved approximation?

Key findings

• An ε-approximation on a compact d*-dimensional manifold with smooth local coordinates is achievable by a sparsely connected ReLU network using O(ε^{-d*/β} log(1/ε)) non-zero parameters.
• The construction builds from local charts, partition of unity, and multiplication networks, combined through network composition rules.
• All network weights can be chosen with absolute value bounded by one, aligning with practical initialization and training constraints.
• For regression with inputs on a manifold, the empirical risk minimizer over the network class achieves a prediction risk of order n^{-2β/(2β+d*)} (up to polylog factors) under suitable choices of depth L and parameter sparsity s.
• The results connect geometric manifold structure (dimension d*) with approximation and statistical rates, highlighting the benefits of exploiting smooth coordinate charts.