[Paper Review] Lipschitz Recurrent Neural Networks

N. Benjamin Erichson, Omri Azencot+3 more|arXiv (Cornell University)|Jun 22, 2020

Anomaly Detection Techniques and Applications51 references32 citations

TL;DR

This paper introduces a continuous-time Lipschitz RNN with a Lipschitz nonlinearity and a symmetric–skew hidden-to-hidden matrix decomposition, proving global exponential stability and showing strong performance on MNIST, TIMIT, and robustness analyses.

ABSTRACT

Viewing recurrent neural networks (RNNs) as continuous-time dynamical systems, we propose a recurrent unit that describes the hidden state's evolution with two parts: a well-understood linear component plus a Lipschitz nonlinearity. This particular functional form facilitates stability analysis of the long-term behavior of the recurrent unit using tools from nonlinear systems theory. In turn, this enables architectural design decisions before experimentation. Sufficient conditions for global stability of the recurrent unit are obtained, motivating a novel scheme for constructing hidden-to-hidden matrices. Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks, including computer vision, language modeling and speech prediction tasks. Finally, through Hessian-based analysis we demonstrate that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs.

Motivation & Objective

• Motivate RNN stability by modeling hidden state dynamics as a continuous-time system with a linear plus Lipschitz nonlinear form.
• Derive sufficient conditions for global exponential stability and propose a symmetric–skew decomposition to construct hidden-to-hidden matrices.
• Demonstrate empirical performance gains on vision (pixel MNIST), language modeling, and speech tasks.
• Show robustness advantages via Hessian-based analysis and perturbation experiments compared to other continuous-time RNNs.

Proposed method

• Propose a continuous-time RNN: dot{h} = A h + tanh(W h + U x + b) with A and W formed as A_{β_A,γ_A} and W_{β_W,γ_W} using a symmetric–skew decomposition parameterization.
• Establish global exponential stability via Lyapunov methods and circle/Kalman–Yakubovitch–Popov criteria under conditions on A^{sym} and W, plus nonlinearity properties.
• Introduce a symmetric-skew decomposition S_{β,γ} = (1−β)(M+M^T) + β(M−M^T) − γ I to bound eigenvalues and control spectral radius.
• Discretize the continuous model with explicit Euler or RK2 integrators for training, ensuring steps stay within stability regions.
• Evaluate on MNIST pixel-by-pixel (ordered and permuted), TIMIT speech prediction, and robustness tests using Hessian metrics and perturbations.

Experimental results

Research questions

• RQ1 Can a Lipschitz-continuous RNN with a linear-plus-nonlinear structure achieve global exponential stability for all states?
• RQ2 How can a symmetric-skew decomposition of hidden matrices balance expressivity with stability to mitigate vanishing/exploding gradients?
• RQ3 Do Lipschitz RNNs outperform state-of-the-art recurrent units on long-range dependency tasks (vision, language, speech) while maintaining robustness to perturbations?
• RQ4 Does higher-order time integration (RK2) improve predictive accuracy over forward Euler within the stability framework?
• RQ5 Are Lipschitz RNNs more robust to input and parameter perturbations as evidenced by Hessian-based analyses?

Key findings

• The Lipschitz RNN achieves global exponential stability under mild spectral conditions on A^{sym} and W and nonlinearity properties.
• Symmetric–skew decomposition enables expressive hidden-to-hidden matrices while satisfying stability criteria, mitigating gradient issues.
• On ordered/permuted MNIST, Lipschitz RNN with Euler RK2 attains competitive to superior accuracy, outperforming several baselines for comparable parameter counts.
• On TIMIT, Lipschitz RNN with Euler/RK2 achieves state-of-the-art or strong performance with around 200K parameters.
• RK2 integration yields higher accuracy than Euler in several tasks, indicating benefits of more accurate numerical integration within the stability framework.
• Hessian-based analysis shows smaller max eigenvalue, lower trace, and better condition number for the Lipschitz RNN, indicating robustness to perturbations and flatter minima.