[Paper Review] On Random Kernels of Residual Architectures

Etai Littwin, Tomer Galanti+1 more|arXiv (Cornell University)|Jan 28, 2020

Advanced Neural Network Applications17 references4 citations

TL;DR

This paper derives finite-width and depth corrections for the Neural Tangent Kernel (NTK) in residual (ResNet) and densely connected (DenseNet) architectures, showing that these networks converge to the infinite-width NTK regime when both width and depth grow simultaneously under proper initialization. Unlike vanilla networks, ResNets and DenseNets achieve bounded NTK variance with minimal parameter complexity—O(L) and O(L²), respectively—enabling effective kernel regression with random gradient features even at finite sizes.

ABSTRACT

We derive finite width and depth corrections for the Neural Tangent Kernel (NTK) of ResNets and DenseNets. Our analysis reveals that finite size residual architectures are initialized much closer to the "kernel regime" than their vanilla counterparts: while in networks that do not use skip connections, convergence to the NTK requires one to fix the depth, while increasing the layers' width. Our findings show that in ResNets, convergence to the NTK may occur when depth and width simultaneously tend to infinity, provided with a proper initialization. In DenseNets, however, convergence of the NTK to its limit as the width tends to infinity is guaranteed, at a rate that is independent of both the depth and scale of the weights. Our experiments validate the theoretical results and demonstrate the advantage of deep ResNets and DenseNets for kernel regression with random gradient features.

Motivation & Objective

• Understand how residual and dense connections affect convergence of the Neural Tangent Kernel (NTK) to its infinite-width limit in finite-width, finite-depth networks.
• Address the limitation of vanilla fully connected networks, where convergence to the NTK regime requires width scaling with depth, leading to high parameter complexity.
• Characterize the rate of convergence of the NTK for residual and dense architectures, particularly focusing on diagonal and off-diagonal terms.
• Validate theoretically derived convergence rates through empirical experiments on MNIST and small UCI datasets using random gradient features.
• Demonstrate that deep ResNets and DenseNets outperform vanilla architectures in kernel regression, even at fixed width, due to improved NTK stability with depth.

Proposed method

• Introduce a forward-backward norm propagation duality for ReLU feedforward networks to analyze the variance of NTK entries.
• Derive finite-width and depth corrections for the NTK of ResNets and DenseNets using recursive analysis of layer-wise activations and gradients.
• Use moment-based analysis to compute the expected squared NTK values and their variance, focusing on E[G(x,x)] and E[G(x,x)²] for diagonal terms.
• Establish bounds on the normalized variance of the NTK via upper and lower bounds on E[G(x,x)²]/E[G(x,x)]², incorporating width n and depth L.
• Apply results from vanilla ReLU networks as a base case, extending them to residual and dense architectures through recursive decomposition of skip connections.
• Conduct empirical evaluation of kernel regression performance using random gradient features G(x;w) as kernel approximations on MNIST and UCI datasets.

Experimental results

Research questions

• RQ1 How does the convergence rate of the NTK to its infinite-width limit depend on width and depth in residual and dense architectures?
• RQ2 Can ResNets and DenseNets achieve NTK convergence when both width and depth grow simultaneously, unlike vanilla networks?
• RQ3 What is the parameter complexity required for finite ResNets and DenseNets to maintain bounded NTK variance compared to vanilla networks?
• RQ4 How does the performance of kernel regression using random gradient features scale with depth in ResNets and DenseNets versus vanilla architectures?
• RQ5 What is the joint distribution behavior of the full NTK matrix in residual and dense networks, and how does it compare to diagonal-only analysis?

Key findings

• For ResNets, convergence to the NTK limit occurs when both width and depth tend to infinity simultaneously, provided proper initialization, with parameter complexity as low as O(L).
• For DenseNets, convergence of the NTK to its infinite-width limit is guaranteed as width increases, with a convergence rate independent of depth and weight scale.
• The normalized variance of the diagonal NTK term G(x,x) in ResNets scales as exp(5m/n + C/n ∑ αl/(1+αl)), which remains bounded when width and depth grow together.
• In DenseNets, the normalized variance is bounded by max{1, 1/(L log²L) · exp(C/n)}, showing that even for fixed width, increasing depth improves kernel approximation quality.
• Empirical results show that for fixed-width ResNets and DenseNets, kernel regression performance using random gradient features improves with depth and approaches the infinite-width limit, in contrast to degradation in vanilla networks.
• The theoretical bounds on NTK variance are validated empirically on MNIST and small UCI datasets, confirming that deep residual and dense architectures maintain stable NTK behavior at finite sizes.