[Paper Review] Is deeper better? It depends on locality of relevant features

T. Mori, Masahito Ueda|arXiv (Cornell University)|May 26, 2020

Neural Networks and Applications5 references4 citations

TL;DR

This paper investigates how network depth affects generalization in overparameterized neural networks using abstract classification tasks with local and global labels. It finds that deeper networks generalize better for local labels due to inherent feature locality from chaotic signal propagation, while shallower networks outperform deeper ones for global labels, highlighting that depth benefits depend critically on feature locality rather than expressivity alone.

ABSTRACT

It has been recognized that a heavily overparameterized artificial neural network exhibits surprisingly good generalization performance in various machine-learning tasks. Recent theoretical studies have made attempts to unveil the mystery of the overparameterization. In most of those previous works, the overparameterization is achieved by increasing the width of the network, while the effect of increasing the depth has remained less well understood. In this work, we investigate the effect of increasing the depth within an overparameterized regime. To gain an insight into the advantage of depth, we introduce local and global labels as abstract but simple classification rules. It turns out that the locality of the relevant feature for a given classification rule plays a key role; our experimental results suggest that deeper is better for local labels, whereas shallower is better for global labels. We also compare the results of finite networks with those of the neural tangent kernel (NTK), which is equivalent to an infinitely wide network with a proper initialization and an infinitesimal learning rate. It is shown that the NTK does not correctly capture the depth dependence of the generalization performance, which indicates the importance of the feature learning rather than the lazy learning.

Motivation & Objective

• To understand why deeper neural networks sometimes generalize better than shallower ones in overparameterized regimes.
• To investigate whether depth provides a general advantage beyond width in overparameterized networks.
• To examine the role of feature locality—local vs. global labels—in determining the optimal network depth for generalization.
• To compare finite-width networks with the Neural Tangent Kernel (NTK) limit, assessing whether lazy learning captures depth-dependent generalization.
• To explore how chaotic signal propagation in deep networks may lead to more localized learned features, even when learning random labels.

Proposed method

• The authors define abstract classification tasks using random inputs and two types of labels: k-local (determined by a few input components) and k-global (determined by a sum over all components).
• They train fully connected feedforward networks of varying depth (L=1 to L=7) and fixed width (H=500) on datasets with N=10^4 samples, using SGD with learning rate η=0.1 and batch size B=50.
• Generalization performance is evaluated via test error, and local stability is measured using s(v), the fraction of test samples stable under local perturbations of strength v.
• The study compares finite-width networks with the Neural Tangent Kernel (NTK) limit, which corresponds to infinite width and infinitesimal learning rates, representing the lazy learning regime.
• Local stability is analyzed by perturbing inputs along individual input dimensions and measuring label invariance, with s(v) quantifying robustness to such perturbations.
• Supplementary experiments validate findings on the Ising model, showing that shallow networks outperform deep ones for global thermodynamic features.

Experimental results

Research questions

• RQ1 Does increasing depth improve generalization performance in overparameterized neural networks, and if so, under what conditions?
• RQ2 How does the locality of relevant features—local versus global—affect the optimal network depth for generalization?
• RQ3 To what extent does the Neural Tangent Kernel (NTK) limit accurately predict the depth dependence of generalization in finite-width networks?
• RQ4 Why do deeper networks sometimes generalize better than shallower ones, despite similar expressivity?
• RQ5 Can chaotic signal propagation in deep networks lead to more localized learned features, even when training on random labels?

Key findings

• Deeper networks achieve significantly lower generalization error than shallower ones for k-local labels, indicating a clear advantage of depth when relevant features are local.
• For k-global labels, shallower networks generalize better than deeper ones, showing that depth is not universally beneficial and can even be detrimental for global feature learning.
• The Neural Tangent Kernel (NTK) limit fails to capture the depth dependence of generalization performance, indicating that feature learning—beyond lazy learning—is essential for generalization in finite-width networks.
• Even when trained on random labels, deeper networks learn more localized features than shallower networks, as evidenced by lower local stability s(v) under perturbations.
• Local stability s(v) is much lower in deep networks than in shallow ones, and s(v) in shallow networks closely matches that of the k-global label, suggesting that shallow networks learn more global representations.
• In a real-world test on the Ising model, shallow networks outperform deep networks in classifying global thermodynamic features, validating the findings in a physically relevant context.