[Paper Review] Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective

Kaixuan Huang, Yuqing Wang+2 more|arXiv (Cornell University)|Feb 14, 2020

Domain Adaptation and Few-Shot LearningComputer Science39 references33 citations

TL;DR

The paper analyzes why deep ResNets generalize better than deep FFNs by comparing their neural tangent kernels (NTKs) in the infinite-width limit and showing that FFN NTKs become degenerate with depth, while ResNet NTKs retain learnable capacity.

ABSTRACT

Deep residual networks (ResNets) have demonstrated better generalization performance than deep feedforward networks (FFNets). However, the theory behind such a phenomenon is still largely unknown. This paper studies this fundamental problem in deep learning from a so-called "neural tangent kernel" perspective. Specifically, we first show that under proper conditions, as the width goes to infinity, training deep ResNets can be viewed as learning reproducing kernel functions with some kernel function. We then compare the kernel of deep ResNets with that of deep FFNets and discover that the class of functions induced by the kernel of FFNets is asymptotically not learnable, as the depth goes to infinity. In contrast, the class of functions induced by the kernel of ResNets does not exhibit such degeneracy. Our discovery partially justifies the advantages of deep ResNets over deep FFNets in generalization abilities. Numerical results are provided to support our claim.

Motivation & Objective

• Motivate understanding of generalization gap between deep ResNets and FFNs.
• Develop NTK-based analysis for deep networks trained end-to-end.
• Compare limiting NTKs of deep FFNs and ResNets under wide/deep regimes.
• Provide nonasymptotic bounds linking network width/depth to NTK behavior.

Proposed method

• Model deep FFNets and ResNets with random Gaussian initializations.
• Derive GP kernels and NTKs for both architectures in the infinite-width limit.
• Normalize NTKs to study limiting behavior as depth grows.
• Prove and/or sketch proofs that FFN NTK degenerates with depth, while ResNet NTK remains learnable.
• Provide nonasymptotic bounds that connect finite-width networks to their limiting NTKs.
• Support theoretical claims with kernel regression experiments on MNIST and CIFAR-10.

Experimental results

Research questions

• RQ1 Do deep FFNs and deep ResNets induce different limiting NTKs as depth goes to infinity?
• RQ2 Is the class of functions induced by the FFN limiting NTK learnable, and is this avoided by the ResNet limiting NTK?
• RQ3 How do width and depth interact to determine the NTK and generalization properties for both architectures?
• RQ4 Can kernel regression with NTK-based kernels reproduce observed generalization differences between FFNs and ResNets?

Key findings

• The FFN NTK converges to a non-informative limiting kernel that yields poor generalization on unseen data.
• The ResNet NTK converges to a learnable limiting kernel that maintains discriminatory power between inputs as depth grows.
• For ResNets with appropriate scaling, the limiting NTK can be depth-invariant, helping explain sustained generalization with very deep models.
• Nonasymptotic bounds show finite-width networks approximate their limiting NTKs under specified width conditions.
• Numerical experiments on MNIST and CIFAR-10 demonstrate FFN-based kernel regressors degrade with depth, while ResNet-based regressors maintain accuracies across depths.