[Paper Review] Correlated Weights in Infinite Limits of Deep Convolutional Neural Networks

Adrià Garriga-Alonso, Mark van der Wilk|arXiv (Cornell University)|Jan 11, 2021

Stochastic Gradient Optimization Techniques29 references4 citations

TL;DR

This paper proposes maintaining spatial correlations in infinitely wide convolutional neural networks by introducing correlated weight priors, rather than relying on architectural changes like mean-pooling. It demonstrates that intermediate levels of weight correlation outperform both independent weights and mean-pooling in infinite-width limits, unifying locally connected and pooled architectures into a continuous spectrum with improved performance.

ABSTRACT

Infinite width limits of deep neural networks often have tractable forms. They have been used to analyse the behaviour of finite networks, as well as being useful methods in their own right. When investigating infinitely wide convolutional neural networks (CNNs), it was observed that the correlations arising from spatial weight sharing disappear in the infinite limit. This is undesirable, as spatial correlation is the main motivation behind CNNs. We show that the loss of this property is not a consequence of the infinite limit, but rather of choosing an independent weight prior. Correlating the weights maintains the correlations in the activations. Varying the amount of correlation interpolates between independent-weight limits and mean-pooling. Empirical evaluation of the infinitely wide network shows that optimal performance is achieved between the extremes, indicating that correlations can be useful.

Motivation & Objective

• To investigate whether the loss of spatial correlation in infinitely wide CNNs is an inevitable consequence of the infinite limit.
• To determine if architectural changes like mean-pooling are the only way to reintroduce spatial correlations in infinite-width CNNs.
• To explore whether correlated weight priors can maintain spatial correlations in the infinite limit without altering network architecture.
• To evaluate empirically whether intermediate levels of weight correlation yield better performance than extreme cases (independent weights or mean-pooling).

Proposed method

• The authors extend existing infinite-width limit frameworks for CNNs by introducing a correlated prior over convolutional filter weights, parameterized by a correlation structure.
• They derive the infinite-width limit of the network's prior distribution under this correlated weight prior, showing that spatial correlations in activations are preserved.
• The method allows continuous interpolation between the independent weight prior (locally connected network) and mean-pooling via a single correlation hyperparameter.
• Theoretical analysis shows that the limiting kernel retains spatial correlation when weights are correlated, unlike in standard independent-prior limits.
• Empirical evaluation is performed on CIFAR-10 and MNIST using exact kernel computations in the infinite-width regime to compare performance across correlation levels.
• The approach replaces discrete architectural choices (e.g., mean-pooling) with a continuous hyperparameter, enabling gradient-based optimization in future work.

Experimental results

Research questions

• RQ1 Is the loss of spatial correlation in infinitely wide CNNs a necessary outcome of the infinite limit, or is it due to the choice of independent weight priors?
• RQ2 Can spatial correlations in activations be preserved in the infinite-width limit without modifying the network architecture?
• RQ3 Does introducing a continuous correlation parameter between weights lead to better performance than the extremes of independent weights or mean-pooling?
• RQ4 Can the discrete architectural choice of mean-pooling be replaced by a continuous correlation parameter in intermediate layers without performance degradation?
• RQ5 Can correlated weight priors in infinite networks inform better priors or initialization schemes in finite-width networks?

Key findings

• The loss of spatial correlation in infinite-width CNNs is not inherent to the infinite limit but results from using independent weight priors.
• Introducing correlated weight priors preserves spatial correlations in the activations, even in the infinite-width limit.
• Empirical results show that optimal performance is achieved at intermediate levels of weight correlation, outperforming both independent weights and mean-pooling.
• Mean-pooling in the final layer can be replaced by an intermediate correlation level without performance degradation, demonstrating the flexibility of the approach.
• The method unifies locally connected networks and mean-pooling architectures into a continuous spectrum, with performance peaking at an intermediate correlation level.
• The framework enables continuous hyperparameter optimization instead of discrete architectural choices, offering a path toward more efficient architecture search.