[Paper Review] The Connection Between Approximation, Depth Separation and Learnability in Neural Networks

Eran Malach, Gilad Yehudai+2 more|arXiv (Cornell University)|Jan 31, 2021

Domain Adaptation and Few-Shot Learning32 references4 citations

TL;DR

This paper establishes a fundamental link between approximation capacity and learnability in deep neural networks, showing that efficient learning via gradient descent or statistical query (SQ) algorithms requires weak approximation by simpler models—such as shallow networks or kernel classes. The key result is that if a function cannot be weakly approximated by such simple classes, it cannot be efficiently learned, even if it is exactly representable by deep networks.

ABSTRACT

Several recent works have shown separation results between deep neural networks, and hypothesis classes with inferior approximation capacity such as shallow networks or kernel classes. On the other hand, the fact that deep networks can efficiently express a target function does not mean that this target function can be learned efficiently by deep neural networks. In this work we study the intricate connection between learnability and approximation capacity. We show that learnability with deep networks of a target function depends on the ability of simpler classes to approximate the target. Specifically, we show that a necessary condition for a function to be learnable by gradient descent on deep neural networks is to be able to approximate the function, at least in a weak sense, with shallow neural networks. We also show that a class of functions can be learned by an efficient statistical query algorithm if and only if it can be approximated in a weak sense by some kernel class. We give several examples of functions which demonstrate depth separation, and conclude that they cannot be efficiently learned, even by a hypothesis class that can efficiently approximate them.

Motivation & Objective

• To investigate the connection between approximation capacity and efficient learnability in deep neural networks.
• To determine whether depth separation implies learnability or if it obstructs efficient learning.
• To formalize the notion of weak approximation and show its necessity for learning via gradient descent and SQ algorithms.
• To analyze known depth-separating functions (e.g., Telgarsky's triangle function, parity functions) and prove they are not efficiently learnable.

Proposed method

• Introduce the concept of weak approximation: a hypothesis class weakly approximates a target function if it performs better than random guessing on average.
• Prove that gradient descent cannot efficiently learn a target class unless 3-layer networks weakly approximate it.
• Establish a characterization: a class is learnable via SQ algorithms if and only if it can be weakly approximated by a kernel class.
• Use rounding techniques and generalization bounds to extend results from specific architectures to general polynomial-size two-hidden-layer networks.
• Analyze known depth-separating functions (e.g., Telgarsky’s function, parity) and show they fail weak approximation by shallow or kernel models.
• Leverage Lipschitz continuity and concentration inequalities to bound generalization error and derive lower bounds on learning error.

Experimental results

Research questions

• RQ1 Can a function that is exactly representable by deep networks still be unlearnable by gradient descent if it cannot be weakly approximated by shallow networks?
• RQ2 Is there a necessary condition for efficient learning via statistical query algorithms, and how does it relate to kernel approximation?
• RQ3 Do known depth-separating functions—such as Telgarsky’s triangle function—fail to be learnable even though they are representable by deep networks?
• RQ4 Can weak approximation by shallow networks or kernel classes be a necessary condition for efficient learning, even when the target is perfectly representable by deeper models?
• RQ5 What is the quantitative trade-off between approximation strength and learning efficiency in deep networks?

Key findings

• A target function cannot be learned by gradient descent on deep networks if it cannot be weakly approximated by 3-layer neural networks.
• A class of functions is learnable by any statistical query algorithm if and only if it can be weakly approximated by a kernel class.
• Telgarsky’s triangle function, which exhibits depth separation, cannot be efficiently learned by gradient descent due to its failure in weak approximation by shallow networks.
• The parity function, which separates deep networks from kernel methods, cannot be learned by efficient SQ algorithms because it cannot be weakly approximated by kernel classes.
• A specific function separating 2-layer and 3-layer networks is shown to be unlearnable by any SQ algorithm, due to lack of weak approximation by shallow or kernel models.
• The paper provides quantitative bounds: for polynomial-size two-hidden-layer networks, the learning error is at least $1 - \frac{6\sqrt{k}R^{2}n^{5/6}}{d(n)^{1/18}}$, showing that poor weak approximation leads to high learning error.