[Paper Review] Adding noise to the input of a model trained with a regularized objective

Salah Rifai, Xavier Glorot+2 more|arXiv (Cornell University)|Apr 16, 2011

Gaussian Processes and Bayesian Inference10 references65 citations

TL;DR

This paper proposes a novel regularization technique that improves generalization in neural networks by adding input noise and explicitly penalizing the L2-norm of the model's Jacobian with respect to input. Using a second-order Taylor expansion of the noisy objective, the method approximates and controls higher-order regularization terms—particularly Hessian penalties—without explicit computation, yielding better robustness and test accuracy with minimal computational overhead.

ABSTRACT

Regularization is a well studied problem in the context of neural networks. It is usually used to improve the generalization performance when the number of input samples is relatively small or heavily contaminated with noise. The regularization of a parametric model can be achieved in different manners some of which are early stopping (Morgan and Bourlard, 1990), weight decay, output smoothing that are used to avoid overfitting during the training of the considered model. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters (Krogh and Hertz, 1991). Using Bishop's approximation (Bishop, 1995) of the objective function when a restricted type of noise is added to the input of a parametric function, we derive the higher order terms of the Taylor expansion and analyze the coefficients of the regularization terms induced by the noisy input. In particular we study the effect of penalizing the Hessian of the mapping function with respect to the input in terms of generalization performance. We also show how we can control independently this coefficient by explicitly penalizing the Jacobian of the mapping function on corrupted inputs.

Motivation & Objective

• To improve generalization performance in neural networks trained on limited or noisy datasets.
• To provide a theoretically grounded method for controlling higher-order regularization terms induced by input noise.
• To enable independent control over Jacobian and Hessian norms in the model's mapping function.
• To reduce computational cost compared to explicit high-order derivative computation while maintaining regularization benefits.

Proposed method

• Uses a Taylor expansion of the loss function around noisy inputs to approximate the effect of input corruption.
• Derives the penalty term induced by input noise as a function of the Jacobian and Hessian of the model's output with respect to input.
• Explicitly penalizes the L2-norm of the Jacobian of the model with respect to input to enforce local invariance under small input perturbations.
• Applies Bishop’s approximation (1995) to relate input noise to an effective regularized objective function.
• Controls regularization strength via hyperparameters that scale the Jacobian and Hessian penalties independently.
• Employs a weak noise limit approximation to derive analytically tractable regularization terms without computing high-order derivatives directly.

Experimental results

Research questions

• RQ1 How does adding noise to the input affect the regularization terms in the objective function?
• RQ2 Can the higher-order derivatives (e.g., Hessian) of the model’s output be effectively regularized without explicit computation?
• RQ3 What is the impact of jointly penalizing the Jacobian and Hessian norms on generalization and robustness?
• RQ4 How does the proposed method compare to standard regularization techniques like weight decay or early stopping in terms of test error?
• RQ5 Can the regularization be tuned independently to control flatness of the loss landscape around training points?

Key findings

• The method achieves a test error of 1.19% on MNIST, outperforming standard MLP (1.82%) and other regularized variants.
• On the MNIST-BINARY dataset, the combined noise and Jacobian regularization reduces error to 1.51%, compared to 2.01% for standard MLP.
• The regularized model shows improved robustness to input corruption, as demonstrated in Figure 2, with lower generalization error under noisy inputs.
• Activation histograms on MNIST show that the regularized model concentrates activations in linear and saturated regimes, indicating flatter and more stable representations.
• Theoretical analysis confirms that input noise induces regularization terms involving the Jacobian and Hessian, which can be controlled independently via explicit penalty terms.
• The approach provides a computationally efficient alternative to explicit high-order derivative computation, with marginal overhead compared to standard training.