[Paper Review] Sharp Rate of Convergence for Deep Neural Network Classifiers under the Teacher-Student Setting

Tianyang Hu, Zuofeng Shang+1 more|arXiv (Cornell University)|Jan 19, 2020

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper establishes a sharp convergence rate of $\widetilde{O}_d(n^{-2/3})$ for deep neural network classifiers under the teacher-student framework, where the Bayes classifier is a ReLU network. The analysis shows that data dimension $d$ only affects the rate logarithmically, offering a theoretical explanation for deep learning's success in high-dimensional classification despite the curse of dimensionality.

ABSTRACT

Classifiers built with neural networks handle large-scale high dimensional data, such as facial images from computer vision, extremely well while traditional statistical methods often fail miserably. In this paper, we attempt to understand this empirical success in high dimensional classification by deriving the convergence rates of excess risk. In particular, a teacher-student framework is proposed that assumes the Bayes classifier to be expressed as ReLU neural networks. In this setup, we obtain a sharp rate of convergence, i.e., $ ilde{O}_d(n^{-2/3})$, for classifiers trained using either 0-1 loss or hinge loss. This rate can be further improved to $ ilde{O}_d(n^{-1})$ when the data distribution is separable. Here, $n$ denotes the sample size. An interesting observation is that the data dimension only contributes to the $\log(n)$ term in the above rates. This may provide one theoretical explanation for the empirical successes of deep neural networks in high dimensional classification, particularly for structured data.

Motivation & Objective

• To provide a theoretical explanation for the empirical success of deep neural networks in high-dimensional classification, particularly in image data.
• To address the gap in statistical understanding of deep learning classifiers, especially under nonparametric settings where traditional smoothness assumptions fail.
• To analyze the convergence rate of excess risk in a teacher-student framework where the optimal decision boundary is a ReLU neural network.
• To investigate whether deep networks can overcome the curse of dimensionality in classification, unlike classical nonparametric methods.
• To extend existing regression-based convergence results to classification by focusing on 0-1 and hinge losses under structured decision boundaries.

Proposed method

• Proposes a teacher-student framework where the Bayes classifier is modeled as a ReLU deep neural network, and the student network is trained to approximate it.
• Analyzes the excess risk of empirical minimizers of 0-1 and hinge losses under this setup, using a non-asymptotic risk decomposition.
• Applies a generalization bound via covering numbers of the function class of student networks, leveraging the sup-norm entropy of deep ReLU networks.
• Uses the Tsybakov noise condition to relate the excess risk to the $\phi$-risk, enabling a sharp rate analysis under weak smoothness assumptions.
• Establishes the convergence rate by verifying conditions in a general excess risk bound lemma, including approximation error, noise condition, and complexity control.
• Derives the rate $\widetilde{O}_d(n^{-2/3})$ by balancing approximation error, estimation error, and network complexity, with $d$ entering only via logarithmic factors.

Experimental results

Research questions

• RQ1 Can deep neural network classifiers achieve fast convergence rates in high-dimensional classification, even when classical nonparametric methods fail?
• RQ2 What is the sharp rate of convergence for excess risk in a teacher-student setting where the Bayes classifier is a ReLU network?
• RQ3 How does the data dimension $d$ affect the convergence rate, and can the curse of dimensionality be mitigated in this setting?
• RQ4 Does the convergence rate improve under separable data distributions, and if so, by how much?
• RQ5 Can the theoretical analysis of deep networks in classification be extended beyond smoothness assumptions, particularly for non-smooth decision boundaries?

Key findings

• The excess risk of the empirical 0-1 loss minimizer converges at a sharp rate of $\widetilde{O}_d(n^{-2/3})$ under the teacher-student setting with ReLU networks.
• When the data distribution is separable, the convergence rate improves to $\widetilde{O}_d(n^{-1})$, indicating faster learning under favorable data conditions.
• The data dimension $d$ only contributes to a logarithmic factor in the convergence rate, suggesting that deep networks can effectively mitigate the curse of dimensionality.
• The analysis holds under weak assumptions, including the Tsybakov noise condition and bounded network weights, making the result robust to real-world data structures.
• The rate $\widetilde{O}_d(n^{-2/3})$ is unimprovable under the given assumptions, establishing it as a sharp bound for this setting.
• The results provide a theoretical foundation for the empirical success of deep learning in high-dimensional image classification, where data dimension is large but generalization remains strong.