[Paper Review] What Can Be Learnt With Wide Convolutional Neural Networks?

Francesco Cagnetta, Alessandro Favero+1 more|arXiv (Cornell University)|Aug 1, 2022

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper studies generalization in infinitely wide deep convolutional neural networks (CNNs) using the Neural Tangent Kernel (NTK) regime. It shows that the NTK spectrum inherits hierarchical structure, enabling CNNs to adapt to the effective dimensionality of target functions: error decay is governed by the size of local receptive fields when functions depend on localized input subsets, but by full input dimension when global dependencies dominate.

ABSTRACT

Understanding how convolutional neural networks (CNNs) can efficiently learn high-dimensional functions remains a fundamental challenge. A popular belief is that these models harness the local and hierarchical structure of natural data such as images. Yet, we lack a quantitative understanding of how such structure affects performance, e.g., the rate of decay of the generalisation error with the number of training samples. In this paper, we study infinitely-wide deep CNNs in the kernel regime. First, we show that the spectrum of the corresponding kernel inherits the hierarchical structure of the network, and we characterise its asymptotics. Then, we use this result together with generalisation bounds to prove that deep CNNs adapt to the spatial scale of the target function. In particular, we find that if the target function depends on low-dimensional subsets of adjacent input variables, then the decay of the error is controlled by the effective dimensionality of these subsets. Conversely, if the target function depends on the full set of input variables, then the error decay is controlled by the input dimension. We conclude by computing the generalisation error of a deep CNN trained on the output of another deep CNN with randomly-initialised parameters. Interestingly, we find that, despite their hierarchical structure, the functions generated by infinitely-wide deep CNNs are too rich to be efficiently learnable in high dimension.

Motivation & Objective

• Understand how the hierarchical and local structure of CNNs affects generalization in high-dimensional function approximation.
• Address the lack of quantitative understanding of how data structure influences generalization error decay rates in deep CNNs.
• Characterize the spectrum of the NTK in infinitely wide deep CNNs to derive generalization bounds.
• Investigate whether hierarchical CNNs can efficiently learn functions with varying degrees of locality or global dependence.
• Determine whether the inductive bias of CNNs mitigates the curse of dimensionality in high-dimensional settings.

Proposed method

• Analyze the Neural Tangent Kernel (NTK) of deep CNNs with non-overlapping patches and no pooling in the infinite-width limit.
• Characterize the spectrum of the NTK by decomposing eigenfunctions into sectors corresponding to each hidden layer’s receptive fields.
• Derive asymptotic behavior of NTK eigenvalues based on the effective dimensionality $ d_{\text{eff}}(l) $ of receptive fields at layer $ l $.
• Apply generalization bounds from kernel ridge regression (Caponnetto & De Vito, 2007) to relate error decay to the effective dimensionality of the target function.
• Use theoretical analysis and empirical validation on synthetic and real datasets (e.g., CIFAR-10) to test predictions.
• Compare learning curves across different input distributions (multisphere, hypercube, Gaussian) and patch types (overlapping vs. non-overlapping).

Experimental results

Research questions

• RQ1 How does the hierarchical structure of deep CNNs influence the spectrum of their NTK in the infinite-width limit?
• RQ2 To what extent can deep CNNs adapt their generalization performance to the spatial scale of the target function?
• RQ3 Does the effective dimensionality of the receptive field at each layer determine the rate of generalization error decay?
• RQ4 Can infinitely wide CNNs efficiently learn high-dimensional functions that depend only on low-dimensional, localized input subsets?
• RQ5 How does the performance of hierarchical CNNs compare to fully connected networks in high-dimensional, structured function approximation tasks?

Key findings

• The spectrum of the NTK in deep CNNs is organized into sector-specific eigenfunctions, each depending only on the receptive field of a given layer.
• Eigenvalues decay asymptotically as a function of polynomial degree and $ d_{\text{eff}}(l) $, the effective dimensionality of the receptive field at layer $ l $, with smaller $ d_{\text{eff}}(l) $ leading to larger eigenvalues.
• Generalization error decays as $ \epsilon(n) \sim n^{-\beta} $, where $ \beta \propto 1/d_{\text{eff}} $ when the target function depends on localized input subsets.
• When the target function depends on the full input dimension, error decay is governed by $ d $, the full input dimension, indicating a curse of dimensionality.
• Despite their hierarchical structure, functions generated by infinitely wide deep CNNs are too rich to be efficiently learnable in high-dimensional input spaces.
• Empirical learning curves on CIFAR-10 and synthetic data confirm theoretical predictions, with deep hierarchical CNNs outperforming fully connected architectures, even under overlapping patches.