[Paper Review] Stable Recurrent Models

J. J. Miller, Moritz Hardt|arXiv (Cornell University)|May 25, 2018

Model Reduction and Neural Networks29 references72 citations

TL;DR

The paper defines stable recurrent models, proves they can be well approximated by feed-forward networks for both inference and gradient-based training, and empirically shows stable variants perform comparably to unstable ones across several sequence tasks.

ABSTRACT

Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks. In this work, we conduct a thorough investigation of stable recurrent models. Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent. Empirically, we demonstrate stable recurrent models often perform as well as their unstable counterparts on benchmark sequence tasks. Taken together, these findings shed light on the effective power of recurrent networks and suggest much of sequence learning happens, or can be made to happen, in the stable regime. Moreover, our results help to explain why in many cases practitioners succeed in replacing recurrent models by feed-forward models.

Motivation & Objective

• Provide a formal definition of stable recurrent models and establish sufficient conditions for stability across common architectures (RNNs and LSTMs).
• Prove that stable recurrent models can be approximated by feed-forward networks for both inference and training via gradient descent.
• Empirically evaluate stable versus unstable models on diverse sequence tasks to assess performance and the practical implications of stability.

Proposed method

• Define stability via contraction: there exists lambda < 1 with ||phi_w(h,x) - phi_w(h',x)|| <= lambda ||h - h'|| for all h,h',x.
• Derive sufficient stability conditions for RNNs with Lipschitz nonlinearity and for LSTMs, including a projection-based training step to enforce stability.
• Prove that under stability, recurrent models and their finite-context (k-step) feed-forward truncations produce similar inferences (y_t ≈ y_t^k).
• Show that with stability, gradient descent on recurrent models closely tracks gradient descent on truncated models, yielding bounded weight differences after N steps.
• Provide a training-time projection scheme (spectral norm projection for RNNs; row-normalization for LSTMs) to enforce stability during learning.

Experimental results

Research questions

• RQ1 Can stable recurrent models be theoretically approximated by finite-context feed-forward networks for both inference and training?
• RQ2 What are practical sufficient conditions to guarantee stability for common recurrent architectures (RNNs and LSTMs)?
• RQ3 Do stable variants retain comparable performance to unstable models across standard sequence tasks?
• RQ4 Does enforcing stability during training inherently limit long-term memory in recurrent models?

Key findings

• Stable recurrent models are contractive in hidden state updates, leading to bounded gradients and improved learnability.
• Under stability, RNNs and LSTMs can be approximated by finite-context feed-forward models for inference, and by gradient descent for training, with provable closeness as context length grows.
• Across tasks (language modeling, polyphonic music, slot-filling), stable models achieve performance comparable to unstable ones; in some cases, a small performance gap remains, particularly for language modeling with LSTMs.
• Data-dependent notions of stability suggest that, on real data, nominally unstable models often operate in a stable regime, mitigating what appears to be a “price of stability.”
• Long-term memory is limited in stable models, implying that similar performance between stable and unstable models often means the task does not require long-range dependencies.