[Paper Review] Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks

Riccardo Aiudi, R. Pacelli+3 more|arXiv (Cornell University)|Jul 21, 2023

Machine Learning and ELMComputer Science3 citations

TL;DR

This paper proposes local kernel renormalization as a mechanism for feature learning in overparametrized convolutional neural networks (CNNs), contrasting it with global renormalization in fully connected (FC) networks. Using a theoretical framework in the proportional limit (P, N₁ → ∞ with α₁ = P/N₁ fixed), it shows that CNNs perform data-dependent, spatially localized kernel adjustments via a matrix of parameters Q̄ᵢⱼ, enabling effective feature learning—unlike FC networks, which only globally rescale a single kernel parameter.

ABSTRACT

Feature learning, or the ability of deep neural networks to automatically learn relevant features from raw data, underlies their exceptional capability to solve complex tasks. However, feature learning seems to be realized in different ways in fully-connected (FC) or convolutional architectures (CNNs). Empirical evidence shows that FC neural networks in the infinite-width limit eventually outperform their finite-width counterparts. Since the kernel that describes infinite-width networks does not evolve during training, whatever form of feature learning occurs in deep FC architectures is not very helpful in improving generalization. On the other hand, state-of-the-art architectures with convolutional layers achieve optimal performances in the finite-width regime, suggesting that an effective form of feature learning emerges in this case. In this work, we present a simple theoretical framework that provides a rationale for these differences, in one hidden layer networks. First, we show that the generalization performance of a finite-width FC network can be obtained by an infinite-width network, with a suitable choice of the Gaussian priors. Second, we derive a finite-width effective action for an architecture with one convolutional hidden layer and compare it with the result available for FC networks. Remarkably, we identify a completely different form of kernel renormalization: whereas the kernel of the FC architecture is just globally renormalized by a single scalar parameter, the CNN kernel undergoes a local renormalization, meaning that the network can select the local components that will contribute to the final prediction in a data-dependent way. This finding highlights a simple mechanism for feature learning that can take place in overparametrized shallow CNNs, but not in shallow FC architectures or in locally connected neural networks without weight sharing.

Motivation & Objective

• To explain why overparametrized CNNs generalize better than fully connected networks (FCNs) despite both being in the infinite-width regime.
• To identify a distinct mechanism for feature learning in CNNs that is absent in FCNs and locally connected networks without weight sharing.
• To formalize the difference between global and local kernel renormalization in one-hidden-layer networks using statistical mechanics.
• To demonstrate that local kernel renormalization in CNNs allows for data-dependent, spatially adaptive feature selection, improving generalization.

Proposed method

• Derives an effective action for a one-hidden-layer CNN in the proportional limit (P, N₁ → ∞ with α₁ = P/N₁ fixed), using statistical field theory.
• Compares the resulting kernel structure in CNNs to that of FCNs, showing that CNNs exhibit a four-index local kernel Kᵢⱼ^{μν} with spatial indices i, j.
• Identifies that CNNs renormalize the kernel via a matrix of parameters Q̄ᵢⱼ, enabling local, data-dependent adjustments, unlike FCNs which use a single scalar Q̄.
• Employs a Gaussian process framework (NNGP) to model the network output and analyze the kernel's evolution during training.
• Validates theoretical predictions with finite-size scaling experiments using CIFAR10 binary classification, training with full-batch gradient descent and ADAM optimizer.
• Uses discretized Langevin dynamics with T = η = 2×10⁻³ to sample the posterior distribution and assess generalization loss over 5×10⁶ epochs.

Experimental results

Research questions

• RQ1 Why do overparametrized CNNs outperform FC networks in generalization despite both being in the infinite-width limit?
• RQ2 What mechanism enables feature learning in CNNs that is absent in FC networks or locally connected networks without weight sharing?
• RQ3 How does the kernel structure in CNNs differ from that in FC networks in the proportional limit?
• RQ4 Can local kernel renormalization in CNNs be formalized as a data-dependent, spatially adaptive adjustment of the kernel matrix?
• RQ5 To what extent does local kernel renormalization improve generalization compared to global renormalization in FC networks?

Key findings

• In the proportional limit, FC networks perform global kernel renormalization via a single scalar parameter Q̄, which rescales the entire kernel matrix Kᵢⱼ^{μν} uniformly.
• CNNs, in contrast, perform local kernel renormalization via a matrix of parameters Q̄ᵢⱼ, enabling data-dependent, spatially localized adjustments to the kernel.
• The local kernel Kᵢⱼ^{μν} captures spatial correlations between image patches, allowing the network to selectively emphasize relevant local features.
• Finite-size scaling experiments confirm that CNNs exhibit a non-trivial mean shift in the kernel matrix (e.g., ΔK̄_{μν}|₁₁ ≈ -1.045 at α₁ = 1), while FCNs show a consistent negative variance trend (σ²(ΔK_{μν}|₀₁) ≈ -2.15 ± 0.01), indicating global tuning.
• Experiments with N₀ = 6400 and N₀ = 1600 confirm robustness of the local renormalization effect across input sizes, with CNNs showing significant kernel shifts (e.g., ΔK̄_{μν}|₁₁ ≈ 0.998 at α₁ = 10) while FCNs remain globally tuned.
• Theoretical and empirical results confirm that local kernel renormalization is a distinct mechanism for feature learning in CNNs, absent in FCNs and locally connected networks without weight sharing.