[Paper Review] Spectral Norm Regularization for Improving the Generalizability of Deep Learning

Yuichi Yoshida, Takeru Miyato|arXiv (Cornell University)|May 31, 2017

Neural Networks and ApplicationsComputer Science28 references218 citations

TL;DR

The paper proposes spectral norm regularization to bound weight matrix spectral norms, aiming to reduce sensitivity to input perturbations and improve generalization, demonstrated across multiple CNN architectures and datasets.

ABSTRACT

We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spectral norm regularization, which penalizes the high spectral norm of weight matrices in neural networks. We provide supportive evidence for the abovementioned hypothesis by experimentally confirming that the models trained using spectral norm regularization exhibit better generalizability than other baseline methods.

Motivation & Objective

• Motivate the study of generalizability through input perturbation sensitivity.
• Propose spectral norm regularization to penalize large spectral norms of weight matrices.
• Show that spectral norm regularization improves generalization and test accuracy, especially with large minibatches.
• Compare spectral norm regularization to weight decay, adversarial training, and Jacobian regularization.
• Provide insights into the relation between input perturbation sensitivity and generalization performance.

Proposed method

• Formulate spectral norm regularization as minimizing empirical risk plus (lambda/2) sum_l sigma(W^l)^2.
• Compute approximate gradients of the spectral norm via a one-iteration power method for efficiency.
• Apply the regularizer to all weight matrices in feed-forward networks (including convolutional layers by reshaping to a matrix).
• Use SGD with Nesterov momentum and grid-searched hyperparameters on several networks and datasets.
• Compare against weight decay and adversarial training, using CIFAR-10/100 and STL-10 datasets.

Experimental results

Research questions

• RQ1 Does restricting the spectral norm of weight matrices improve generalization to unseen data?
• RQ2 How does spectral norm regularization compare with conventional regularizers (weight decay, adversarial training) in terms of test accuracy and generalization gap?
• RQ3 Is insensitivity to test-data perturbation a key predictor of generalization performance in deep networks?
• RQ4 What is the effect of spectral norm regularization on the spectrum of weight matrices?
• RQ5 How does spectral norm regularization perform under small vs. large minibatch regimes?

Key findings

• Spectral norm regularization yields the best test accuracy in several models, especially in the large minibatch regime.
• Across models and datasets, spectral norm regularization consistently achieves the smallest generalization gap.
• The gradient norm with respect to test data correlates with generalization gap, and spectral norm regularization reduces sensitivity to input perturbations.
• Regularization with spectral norm tends to flatten the singular value spectrum of weight matrices, reducing the largest singular value.
• Maximum Hessian eigenvalue of the training loss is not a reliable predictor of generalization when using spectral norm regularization.
• Spectral norm regularization preserves model capacity better than plain weight decay while still improving generalization.