[Paper Review] Explicit Regularisation in Gaussian Noise Injections

Alexander Camuto, Matthew Willetts+3 more|arXiv (Cornell University)|Jul 14, 2020

Sparse and Compressive Sensing Techniques48 references36 citations

TL;DR

The paper derives an explicit regulariser from Gaussian noise injections in neural networks, showing it penalises high-frequency components (especially near the output) and leads to calibrated classifiers with larger margins.

ABSTRACT

We study the regularisation induced in neural networks by Gaussian noise injections (GNIs). Though such injections have been extensively studied when applied to data, there have been few studies on understanding the regularising effect they induce when applied to network activations. Here we derive the explicit regulariser of GNIs, obtained by marginalising out the injected noise, and show that it penalises functions with high-frequency components in the Fourier domain; particularly in layers closer to a neural network's output. We show analytically and empirically that such regularisation produces calibrated classifiers with large classification margins.

Motivation & Objective

• Motivate and understand the regularisation effect induced by Gaussian noise injections (GNIs) applied to network activations.
• Derive an analytic explicit regulariser by marginalising out the injected noise.
• Show the regulariser relates to Sobolev spaces and the Fourier domain.
• Demonstrate that GNIs yield larger classification margins and improved calibration.
• Provide analytic and empirical evidence across regression and classification settings.

Proposed method

• Model the effect of isotropic Gaussian noise injected at each hidden layer on the loss via a Taylor expansion of layer activations.
• Marginalise the injected noise to obtain an explicit regulariser term, R, that dominates higher-order remainder terms (Theorem 1).
• Express R in regression and classification cases, linking it to Jacobians and Hessians of the network (Equations (11)–(14)).
• Connect R to Sobolev spaces and Fourier transforms to show a bias toward low-frequency (smooth) functions (Theorems 2 and related discussion).
• Demonstrate a recursive, layer-wise regularisation where early layers contribute more to high-frequency penalties, leading to progressively smoother deeper layers (layer-wise discussion).
• Provide empirical evidence showing GNIs and R induce similar training dynamics and lower-frequency learning (Figure 3 and Figure 4).

Experimental results

Research questions

• RQ1 What explicit regularisation is induced by Gaussian Noise Injections when applied to neural network activations?
• RQ2 How is this regulariser connected to Sobolev spaces and the Fourier spectrum of the learned functions?
• RQ3 Does the regulariser promote smoother (lower-frequency) functions and larger classification margins?
• RQ4 Do GNIs improve model calibration and robustness in regression and classification settings?

Key findings

• The added regulariser from GNIs, R, is positive and dominates the higher-order remainder terms in practice.
• R is connected to the Sobolev norm and to Fourier-domain penalties that suppress high-frequency components.
• Regularisation by GNIs is stronger for layers closer to the output, promoting progressively lower-frequency representations.
• Empirically, models trained with GNIs and R exhibit similar training trajectories and show reduced sensitivity and better calibration.
• In regression, the explicit regulariser reduces to a sum of Frobenius norms of layer Jacobians weighted by noise variances (Equation (11)).
• In classification, R involves the Hessian of the loss and remains a positive, frequency-penalising term (Equations (13)-(14)).