[Paper Review] Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks

Hao Liu, Minshuo Chen+2 more|arXiv (Cornell University)|Sep 7, 2021

Computational Physics and Python Applications4 citations

TL;DR

This paper establishes theoretical guarantees for convolutional residual networks (ConvResNets) in approximating Besov functions and performing binary classification on low-dimensional manifolds embedded in high-dimensional space. By exploiting the intrinsic dimension $d$ rather than the ambient dimension $D$, the authors prove that ConvResNets achieve an excess risk of order $n^{-\frac{s}{2s+2(s\vee d)}}$, demonstrating sample complexity dependence on $d$ and explaining the empirical success of deep learning on high-dimensional data with low-dimensional structure.

ABSTRACT

Most of existing statistical theories on deep neural networks have sample complexities cursed by the data dimension and therefore cannot well explain the empirical success of deep learning on high-dimensional data. To bridge this gap, we propose to exploit low-dimensional geometric structures of the real world data sets. We establish theoretical guarantees of convolutional residual networks (ConvResNet) in terms of function approximation and statistical estimation for binary classification. Specifically, given the data lying on a $d$-dimensional manifold isometrically embedded in $\mathbb{R}^D$, we prove that if the network architecture is properly chosen, ConvResNets can (1) approximate Besov functions on manifolds with arbitrary accuracy, and (2) learn a classifier by minimizing the empirical logistic risk, which gives an excess risk in the order of $n^{-\frac{s}{2s+2(s\vee d)}}$, where $s$ is a smoothness parameter. This implies that the sample complexity depends on the intrinsic dimension $d$, instead of the data dimension $D$. Our results demonstrate that ConvResNets are adaptive to low-dimensional structures of data sets.

Motivation & Objective

• To address the gap between empirical success and theoretical understanding of deep learning in high-dimensional data.
• To explain why deep learning generalizes well despite the curse of dimensionality in existing theories.
• To develop a statistical theory for ConvResNets that accounts for low-dimensional geometric structures in real-world data.
• To establish function approximation and classification error bounds that depend on the intrinsic dimension $d$ rather than ambient dimension $D$.

Proposed method

• Theoretical analysis of ConvResNet architecture with skip connections and convolutional layers on $d$-dimensional manifolds embedded in $\mathbb{R}^D$.
• Use of Besov function spaces to model irregular or non-smooth functions common in real data.
• Construction of a deep network $\bar{f}_{\phi,n}$ via residual blocks and convolutional layers to approximate target functions on the manifold.
• Bounding the covering number of the network class using layer-wise complexity, leading to generalization error control.
• Derivation of excess risk bounds via empirical risk minimization of logistic loss, leveraging manifold structure.
• Application of concentration inequalities and approximation theory for manifolds to control estimation and approximation errors.

Experimental results

Research questions

• RQ1 Can ConvResNets approximate Besov functions on low-dimensional manifolds with arbitrary accuracy?
• RQ2 Does the sample complexity of ConvResNet-based binary classification depend on the intrinsic dimension $d$ rather than the ambient dimension $D$?
• RQ3 Can the generalization error of ConvResNets be bounded in terms of the smoothness parameter $s$ and manifold dimension $d$?
• RQ4 How does the network architecture, particularly skip connections and convolution, contribute to approximation and estimation on manifolds?
• RQ5 Is the theoretical excess risk bound adaptive to the intrinsic geometry of data, even when data is high-dimensional?

Key findings

• ConvResNets can approximate any Besov function on a $d$-dimensional manifold with arbitrary accuracy when the architecture is properly chosen.
• The excess risk of ConvResNet for binary classification scales as $n^{-\frac{s}{2s+2(s\vee d)}}$, where $s$ is the smoothness parameter and $d$ is the intrinsic dimension.
• The sample complexity depends on the intrinsic dimension $d$ rather than the ambient dimension $D$, avoiding the curse of dimensionality.
• The generalization error bound is derived using covering number bounds on the network class, which scale with $d$ and $s$, not $D$.
• The theory demonstrates that ConvResNets are adaptive to low-dimensional data structures, explaining their empirical success on high-dimensional data.
• The theoretical framework supports the use of fixed-width networks with skip connections for efficient and accurate learning on manifolds.