[Paper Review] A Basic Recurrent Neural Network Model

Fathi M. Salem|arXiv (Cornell University)|Dec 29, 2016

Neural Networks and Applications4 references4 citations

TL;DR

This paper proposes a Basic Recurrent Neural Network (bRNN) model that ensures bounded solutions and fast dynamics through a stable linear term with a fixed matrix, avoiding vanishing/exploding gradients. It uses a constrained optimization framework with Lagrange multipliers and Calculus of Variations to derive principled gradient descent, enabling unified treatment of supervised and unsupervised loss functions on outputs and hidden states.

ABSTRACT

We present a model of a basic recurrent neural network (or bRNN) that includes a separate linear term with a slightly "stable" fixed matrix to guarantee bounded solutions and fast dynamic response. We formulate a state space viewpoint and adapt the constrained optimization Lagrange Multiplier (CLM) technique and the vector Calculus of Variations (CoV) to derive the (stochastic) gradient descent. In this process, one avoids the commonly used re-application of the circular chain-rule and identifies the error back-propagation with the co-state backward dynamic equations. We assert that this bRNN can successfully perform regression tracking of time-series. Moreover, the "vanishing and exploding" gradients are explicitly quantified and explained through the co-state dynamics and the update laws. The adapted CoV framework, in addition, can correctly and principally integrate new loss functions in the network on any variable and for varied goals, e.g., for supervised learning on the outputs and unsupervised learning on the internal (hidden) states.

Motivation & Objective

• Address the persistent challenges of vanishing and exploding gradients in simple RNNs without relying on complex gating mechanisms.
• Develop a theoretically grounded, stable recurrent network architecture that ensures bounded input-bounded output (BIBO) stability.
• Provide a unified framework for training RNNs using constrained optimization and Calculus of Variations, enabling integration of diverse loss functions.
• Enable both supervised learning on outputs and unsupervised learning on internal states through a flexible, principled parameter update mechanism.
• Demonstrate that the bRNN can successfully perform regression tracking of continuous time-series data with guaranteed dynamic stability.

Proposed method

• Formulates the bRNN as a discrete-time nonlinear dynamical system with a stable linear term using a fixed matrix to ensure bounded solutions.
• Applies the Lagrange Multiplier Constrained Optimization (CLM) technique to derive the gradient descent update laws without reapplying the chain rule.
• Uses the Calculus of Variations (CoV) to derive co-state backward dynamics, identifying error backpropagation with costate equations.
• Defines split boundary conditions: initial state $x_0$ and final co-state $\lambda_N = \partial\phi/\partial x_N$, enabling forward state and backward co-state propagation.
• Introduces a general loss function $L^k$ that can include supervised (output error), unsupervised (state sparsity, entropy), and regularization terms.
• Derives closed-form gradient updates for all parameters ($U, W, b, V, D, c$) using Jacobian derivatives and co-state propagation, including regularization terms.

Experimental results

Research questions

• RQ1 Can a simple recurrent network architecture be designed to avoid the vanishing and exploding gradient problems without additional gating units?
• RQ2 How can the principles of constrained optimization and Calculus of Variations be systematically applied to derive stable and principled gradient updates in RNNs?
• RQ3 To what extent can the framework support both supervised and unsupervised learning objectives within a single RNN training process?
• RQ4 What is the role of the fixed linear matrix in ensuring BIBO stability and fast dynamic response in the bRNN model?
• RQ5 How can the co-state dynamics explicitly quantify and explain the behavior of gradients during backpropagation in recurrent networks?

Key findings

• The bRNN model achieves BIBO stability without requiring gating mechanisms, reducing parameter count compared to LSTM or GRU architectures.
• The vanishing and exploding gradient phenomena are explicitly quantified through co-state dynamics, showing their dependence on the eigenvalues of the state transition matrix.
• The co-state backward dynamics naturally yield the error backpropagation process, identifying it as a solution to the costate equation.
• The framework allows seamless integration of diverse loss functions—supervised on outputs and unsupervised on hidden states—without complex re-derivations.
• Parameter updates are derived in closed form using CoV and CLM, with explicit expressions for $\Delta U_k$, $\Delta W_k$, $\Delta b_k$, $\Delta V_k$, $\Delta D_k$, and $\Delta c_k$.
• The final co-state is computed as $\lambda_N = (\sigma_N')^T V_N^T e_N$, linking the terminal error directly to the backward pass.