[Paper Review] Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee

Wei Hu, Zhiyuan Li+1 more|arXiv (Cornell University)|May 27, 2019

Machine Learning and AlgorithmsComputer Science37 references45 citations

TL;DR

The paper introduces two simple regularization techniques (distance to initialization and per-sample auxiliary variables) for training on noisily labeled data and proves they yield NTK-based kernel ridge regression with generalization guarantees, supported by experiments on MNIST and CIFAR-10.

ABSTRACT

Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data. Such over-fitting ability hinders generalization when mislabeled training examples are present. On the other hand, simple regularization methods like early-stopping can often achieve highly nontrivial performance on clean test data in these scenarios, a phenomenon not theoretically understood. This paper proposes and analyzes two simple and intuitive regularization methods: (i) regularization by the distance between the network parameters to initialization, and (ii) adding a trainable auxiliary variable to the network output for each training example. Theoretically, we prove that gradient descent training with either of these two methods leads to a generalization guarantee on the clean data distribution despite being trained using noisy labels. Our generalization analysis relies on the connection between wide neural network and neural tangent kernel (NTK). The generalization bound is independent of the network size, and is comparable to the bound one can get when there is no label noise. Experimental results verify the effectiveness of these methods on noisily labeled datasets.

Motivation & Objective

• Motivate the need for regularization when training over-parameterized networks on noisily labeled data.
• Propose two simple regularization methods that limit overfitting to noisy labels.
• Establish theoretical generalization guarantees via the NTK framework for wide networks.
• Show empirical evidence that the proposed methods match or exceed early stopping on noisily labeled data.

Proposed method

• Regularization by distance to initialization (RDI) adds a penalty lambda^2/2 * ||theta - theta(0)||^2 to the loss.
• Auxiliary-variable (AUX) method introduces per-sample trainable variables b_i and uses f(theta, x_i) + lambda*b_i to fit each noisy label.
• In the wide-network (NTK) regime, both methods yield the kernel ridge regression solution with NTK: f*(x) = k(x, X)^T (k(X, X) + lambda^2 I)^{-1} tilde_y.
• Equivalence: gradient descent on the regularized objectives yields the same dynamics as linearized NTK regression.
• Extension to multi-output settings shows each output follows the corresponding NTK ridge regression form.
• Theoretical analysis relies on the NTK approximation f(theta, x) ≈ f(theta(0), x) + phi(x)^T (theta - theta(0)) and small initialization.

Experimental results

Research questions

• RQ1 Can simple regularization schemes prevent overfitting to noisy labels in over-parameterized networks?
• RQ2 Do RDI and AUX regularizations lead to generalization guarantees on clean data distributions under noisy training labels?
• RQ3 How do these methods relate to the NTK and kernel ridge regression in wide networks?
• RQ4 Are the proposed methods practically effective compared to early stopping on noisily labeled data?
• RQ5 Do results extend to multi-class classification and deeper networks?

Key findings

• Both RDI and AUX regularization lead gradient descent to kernel ridge regression with NTK in the wide-net regime.
• The generalization bound on the clean data distribution is comparable to the bound with no label noise and depends on the (unobserved) clean labels.
• Empirically, RDI and AUX achieve test accuracies similar to or better than early stopping across MNIST and CIFAR-10 under various label-noise levels.
• Weight movement during training is reduced by the regularizers, aligning with NTK regime expectations.
• AUX demonstrates strong performance on CIFAR-10 with high noise, achieving competitive results across MSE and cross-entropy losses.