[Paper Review] Equilibrated Recurrent Neural Network: Neuronal Time-Delayed Self-Feedback Improves Accuracy and Stability

Ziming Zhang, Anil Kag+2 more|arXiv (Cornell University)|Mar 2, 2019

Neural Networks and Applications46 references4 citations

TL;DR

This paper proposes the Equilibrated Recurrent Neural Network (ERNN), which integrates time-delayed self-feedback loops inspired by neuronal autapses to stabilize hidden state dynamics, accelerate training convergence, and improve test accuracy. By solving fixed-point conditions via an inexact Newton method, ERNN achieves state-of-the-art performance on multiple benchmarks with faster training and better generalization than RNNs, FastRNN, and FastGRNN-LSQ.

ABSTRACT

We propose a novel {\it Equilibrated Recurrent Neural Network} (ERNN) to combat the issues of inaccuracy and instability in conventional RNNs. Drawing upon the concept of autapse in neuroscience, we propose augmenting an RNN with a time-delayed self-feedback loop. Our sole purpose is to modify the dynamics of each internal RNN state and, at any time, enforce it to evolve close to the equilibrium point associated with the input signal at that time. We show that such self-feedback helps stabilize the hidden state transitions leading to fast convergence during training while efficiently learning discriminative latent features that result in state-of-the-art results on several benchmark datasets at test-time. We propose a novel inexact Newton method to solve fixed-point conditions given model parameters for generating the latent features at each hidden state. We prove that our inexact Newton method converges locally with linear rate (under mild conditions). We leverage this result for efficient training of ERNNs based on backpropagation.

Motivation & Objective

• Address the instability and slow convergence of conventional RNNs during training, particularly due to vanishing or exploding gradients.
• Improve test-time generalization by learning more discriminative latent features through stabilized hidden state dynamics.
• Introduce a biologically inspired self-feedback mechanism—autapse-like time-delayed connections—to control and equilibrate internal state evolution.
• Develop an efficient training framework based on fixed-point solving that enables backpropagation through the equilibrium dynamics.
• Demonstrate that self-feedback can outperform traditional gating mechanisms in both accuracy and training speed.

Proposed method

• Augment each RNN hidden state with a time-delayed self-feedback connection, modifying the update rule to enforce evolution toward an equilibrium point for the current input.
• Formulate the hidden state update as a fixed-point equation: $\mathbf{h}_t = \tanh(\mathbf{h}_t + \mathbf{V}\mathbf{h}_{t-1} + \mathbf{W}\mathbf{x}_t + \mathbf{b})$, where $\mathbf{h}_t$ is implicitly defined by the fixed point.
• Propose a novel inexact Newton method to solve the fixed-point condition efficiently, with local linear convergence under mild conditions.
• Use upsampled inputs with constant interpolation to allow each input to reach a set-point over time, enabling stable equilibrium-seeking behavior.
• Train the ERNN using backpropagation through the fixed-point solver, treating the equilibrium state as a differentiable function of parameters.
• Learn a learnable weight $\eta$ for each timestep to control the feedback strength, allowing both positive and negative feedback for better adaptation.

Experimental results

Research questions

• RQ1 Can time-delayed self-feedback in RNNs stabilize hidden state transitions and improve training convergence?
• RQ2 Does incorporating autapse-inspired feedback lead to better generalization and higher test accuracy compared to standard RNNs and gated networks?
• RQ3 Can an inexact Newton method efficiently solve the fixed-point condition for equilibrium states, enabling differentiable training?
• RQ4 How does the learned feedback strength $\eta$ vary across timesteps and datasets, and does it enable better performance than fixed or positive-only feedback?
• RQ5 Can ERNN achieve faster training and higher accuracy than state-of-the-art RNNs like FastRNN and FastGRNN-LSQ?

Key findings

• On the Google-30 dataset, ERNN achieved 2.50% higher accuracy than FastRNN while training ~3× faster.
• On the Pixel-MNIST dataset, ERNN surpassed FastRNN by 3.16% in accuracy and trained 6× faster.
• ERNN(K=2) achieved 96.33% accuracy on HAR-2, outperforming FastGRNN-LSQ by 1% with similar training time.
• The inexact Newton method demonstrated local linear convergence, enabling efficient and stable fixed-point solving during backpropagation.
• ERNN showed significantly faster convergence in training loss compared to FastRNN, with lower final losses, indicating improved optimization dynamics.
• Learned $\eta$ values were small and often negative, suggesting that adaptive feedback, including inhibition, is crucial for performance, and patterns of $\eta$ were consistent across different $k$ values for the same dataset.