[Paper Review] Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization

Jiong Zhang, Qi Lei+1 more|arXiv (Cornell University)|Mar 25, 2018

Matrix Theory and AlgorithmsComputer Science63 citations

TL;DR

The paper introduces Spectral-RNN, an efficient SVD-based parameterization of weight matrices that explicitly controls singular values to stabilize gradients in RNNs (and generalizes to non-square matrices), improving training speed and generalization.

ABSTRACT

Vanishing and exploding gradients are two of the main obstacles in training deep neural networks, especially in capturing long range dependencies in recurrent neural networks~(RNNs). In this paper, we present an efficient parametrization of the transition matrix of an RNN that allows us to stabilize the gradients that arise in its training. Specifically, we parameterize the transition matrix by its singular value decomposition(SVD), which allows us to explicitly track and control its singular values. We attain efficiency by using tools that are common in numerical linear algebra, namely Householder reflectors for representing the orthogonal matrices that arise in the SVD. By explicitly controlling the singular values, our proposed Spectral-RNN method allows us to easily solve the exploding gradient problem and we observe that it empirically solves the vanishing gradient issue to a large extent. We note that the SVD parameterization can be used for any rectangular weight matrix, hence it can be easily extended to any deep neural network, such as a multi-layer perceptron. Theoretically, we demonstrate that our parameterization does not lose any expressive power, and show how it controls generalization of RNN for the classification task. %, and show how it potentially makes the optimization process easier. Our extensive experimental results also demonstrate that the proposed framework converges faster, and has good generalization, especially in capturing long range dependencies, as shown on the synthetic addition and copy tasks, as well as on MNIST and Penn Tree Bank data sets.

Motivation & Objective

• Motivate and address vanishing and exploding gradients in deep networks, especially RNNs.
• Propose an SVD-based weight parameterization that retains expressive power while enabling spectral control.
• Develop Spectral-RNN that constrains singular values to improve gradient stability without increasing complexity.
• Extend the SVD parameterization to non-square weight matrices for MLPs and residual networks.
• Provide theoretical generalization insights under spectral constraints and validate empirically across tasks.

Proposed method

• Parameterize weight matrices W by their SVD W = U Σ V^T and represent U and V compactly using products of Householder reflectors.
• Maintain W in SVD form during training to explicitly track and constrain singular values.
• Constrain singular values with a parameterization that keeps them near 1, using a sigmoid-based update scheme for σ.
• Compute forward and backward passes efficiently as products of Householder reflections, preserving O(n) or near-linear complexity per layer.
• Extend the parameterization to non-square W for MLPs by expressing W via reduced SVD and a compact Householder representation.
• Apply the framework to Spectral-RNN, non-square weight matrices, and explore computational costs comparable to vanilla RNNs.

Experimental results

Research questions

• RQ1 Can gradient vanishing/exploding be mitigated without sacrificing network expressivity?
• RQ2 Does explicit spectral control via SVD parameterization improve optimization, generalization, and ability to model long-range dependencies?
• RQ3 Can the SVD-based parameterization be efficiently applied to non-square weight matrices in MLPs and other architectures?
• RQ4 What theoretical guarantees exist relating spectral constraints to generalization in RNNs?
• RQ5 How does Spectral-RNN perform empirically on synthetic tasks and standard datasets compared to baselines such as RNN, oRNN, and LSTM?

Key findings

• Spectral-RNN shows improved gradient stability and faster convergence compared to Vanilla RNN, IRNN, oRNN, and LSTM on synthetic addition and copy tasks, especially as depth increases.
• Spectral-RNN achieves higher accuracy on pixel-MNIST (97.7% with 128 hidden units) and competitive performance on permuted-MNIST, outperforming several baselines.
• On Penn Tree Bank, Spectral-RNN with 1- and 2-layer configurations attains lower train perplexity and comparable or better test perplexity with fewer parameters than LSTM.
• Theoretical result: generalization gap for RNNs under margin loss is bounded by a factor that grows with the spectral norm of transition matrices, and constraining singular values improves generalization.
• Extending to non-square matrices, the SVD parameterization retains expressive power and enables application to MLPs and residual networks while maintaining similar computational costs.
• Empirical results indicate Spectral-RNN preserves long-range dependency modeling better than baselines, with robust gradients across training.