[Paper Review] When Recurrent Models Don't Need To Be Recurrent.

J. J. Miller, Moritz Hardt|arXiv (Cornell University)|May 25, 2018

Domain Adaptation and Few-Shot Learning6 references29 citations

TL;DR

This paper demonstrates that stable recurrent neural networks can be effectively approximated by feed-forward networks during both inference and training via gradient descent. Under a natural stability condition, the theoretical and empirical results show that recurrent models are not inherently necessary for sequence learning, challenging the conventional reliance on recurrence.

ABSTRACT

We prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent. Our result applies to a broad range of non-linear recurrent neural networks under a natural stability condition, which we observe is also necessary. Complementing our theoretical findings, we verify the conclusions of our theory on both real and synthetic tasks. Furthermore, we demonstrate recurrent models satisfying the stability assumption of our theory can have excellent performance on real sequence learning tasks.

Motivation & Objective

• To investigate whether stable recurrent neural networks can be effectively replaced by feed-forward networks.
• To identify the conditions under which recurrence is not essential for sequence modeling.
• To establish a theoretical foundation for approximating recurrent dynamics with feed-forward architectures.
• To validate the theory empirically on synthetic and real-world sequence learning tasks.

Proposed method

• Theoretical analysis focuses on recurrent neural networks under a stability condition, ensuring bounded dynamics.
• The paper proves that stable RNNs can be uniformly approximated by feed-forward networks of comparable width.
• The approximation is shown to hold for both inference and training via gradient descent.
• The stability condition is formalized as a Lipschitz constraint on the recurrent transition function.
• Empirical validation is conducted on synthetic tasks and real sequence learning benchmarks.
• Performance comparisons are made between stable RNNs and their feed-forward approximations.

Experimental results

Research questions

• RQ1 Under what conditions can feed-forward networks approximate recurrent networks without significant performance loss?
• RQ2 Is the stability condition both sufficient and necessary for the equivalence between recurrent and feed-forward models?
• RQ3 Can stable RNNs achieve strong performance on real sequence learning tasks without requiring recurrence?
• RQ4 How well do feed-forward networks approximate the dynamics of stable RNNs during training with gradient descent?
• RQ5 Does the theoretical approximation hold in practice across diverse sequence learning scenarios?

Key findings

• Stable recurrent neural networks can be uniformly approximated by feed-forward networks of similar width.
• The stability condition is both sufficient and necessary for the approximation to hold.
• Empirical results confirm that feed-forward networks match the performance of stable RNNs on synthetic and real sequence tasks.
• Training via gradient descent is preserved in the feed-forward approximation, maintaining optimization dynamics.
• Recurrent models satisfying the stability assumption achieve strong performance, indicating recurrence is not inherently required.