[Paper Review] Theoretical Analysis of Inductive Biases in Deep Convolutional Networks

Zihao Wang, Lei Wu|arXiv (Cornell University)|May 15, 2023

Stochastic Gradient Optimization Techniques7 citations

TL;DR

This paper provides a theoretical analysis of inductive biases in deep convolutional neural networks (CNNs), demonstrating that a depth of 𝒪(log d) suffices for universality and that CNNs require only 𝒪̃(log²d) samples to learn sparse functions. The work isolates the roles of weight sharing and locality, proving CNNs outperform locally-connected and fully-connected networks by orders of magnitude in sample efficiency due to symmetry-breaking via these inductive biases.

ABSTRACT

In this paper, we provide a theoretical analysis of the inductive biases in convolutional neural networks (CNNs). We start by examining the universality of CNNs, i.e., the ability to approximate any continuous functions. We prove that a depth of $\mathcal{O}(\log d)$ suffices for deep CNNs to achieve this universality, where $d$ in the input dimension. Additionally, we establish that learning sparse functions with CNNs requires only $\widetilde{\mathcal{O}}(\log^2d)$ samples, indicating that deep CNNs can efficiently capture {\em long-range} sparse correlations. These results are made possible through a novel combination of the multichanneling and downsampling when increasing the network depth. We also delve into the distinct roles of weight sharing and locality in CNNs. To this end, we compare the performance of CNNs, locally-connected networks (LCNs), and fully-connected networks (FCNs) on a simple regression task, where LCNs can be viewed as CNNs without weight sharing. On the one hand, we prove that LCNs require $Ω(d)$ samples while CNNs need only $\widetilde{\mathcal{O}}(\log^2d)$ samples, highlighting the critical role of weight sharing. On the other hand, we prove that FCNs require $Ω(d^2)$ samples, whereas LCNs need only $\widetilde{\mathcal{O}}(d)$ samples, underscoring the importance of locality. These provable separations quantify the difference between the two biases, and the major observation behind our proof is that weight sharing and locality break different symmetries in the learning process.

Motivation & Objective

• To theoretically understand why CNNs outperform fully-connected networks (FCNs) in vision tasks despite similar representational capacity.
• To isolate and quantify the individual contributions of weight sharing and locality in CNNs by comparing against locally-connected networks (LCNs).
• To establish the universality of deep CNNs with minimal depth and to analyze sample complexity for learning sparse functions.
• To demonstrate that multichanneling and downsampling are essential for achieving logarithmic depth universality and efficient learning of long-range correlations.

Proposed method

• Proved universality of deep CNNs using a novel combination of multichanneling and downsampling, showing that downsampling exponentially expands the receptive field.
• Used multichanneling to preserve information during spatial reduction, ensuring no loss during depth progression.
• Formulated a synthetic regression task to compare CNNs, LCNs, and FCNs, enabling separation of weight sharing and locality effects.
• Applied statistical learning theory to derive sample complexity bounds, using concentration inequalities and Frobenius norm bounds on parameter differences.
• Employed Cauchy-Schwarz and Gaussian tail bounds to derive lower bounds on generalization error, linking parameter distance to function distance.
• Established a lower bound on depth by showing that without downsampling, CNNs require Ω(d) depth for universality, proving downsampling's necessity.

Experimental results

Research questions

• RQ1 Can deep CNNs achieve universality with depth logarithmic in input dimension d, and what structural components enable this?
• RQ2 How does the sample complexity of learning sparse functions scale in CNNs, and is it near-optimal?
• RQ3 What is the individual contribution of weight sharing versus locality to sample efficiency in deep networks?
• RQ4 Why do CNNs outperform LCNs and FCNs in learning tasks requiring long-range correlations?
• RQ5 Can the synergy of multichanneling, downsampling, weight sharing, and locality be theoretically linked to improved generalization?

Key findings

• A depth of 𝒪(log d) is sufficient for deep CNNs to universally approximate any continuous function, a significant improvement over prior works requiring Ω(d) depth.
• Without downsampling, CNNs require Ω(d) depth for universality, proving that downsampling is essential for logarithmic depth universality.
• Learning sparse functions with CNNs requires only 𝒪̃(log²d) samples, which is nearly optimal compared to the information-theoretic lower bound of Ω(log d).
• CNNs can learn any k-sparse function with only 𝒪(k² log d) total parameters, a dramatic improvement over LASSO, which requires Ω(d) parameters in the linear case.
• LCNs require Ω(d) samples to learn the same tasks, while CNNs need only 𝒪̃(log²d) samples, proving that weight sharing is crucial for sample efficiency.
• FCNs require Ω(d²) samples, while LCNs need only 𝒪̃(d) samples, demonstrating that locality (small filters) is essential for reducing sample complexity.