[Paper Review] Online learning of long-range dependencies

Nicolas Zucchet, R. R. Meier+3 more|arXiv (Cornell University)|May 25, 2023

Advanced Memory and Neural Computing4 citations

TL;DR

This paper proposes a novel online learning algorithm for recurrent neural networks that achieves exact gradient computation with only double the memory and computation of inference by leveraging independent recurrent modules in multi-layer architectures. The method enables strong performance on long-range dependency tasks—matching BPTT performance on sequence lengths up to 4,000—demonstrating that modularity simplifies online credit assignment and offers a biologically plausible learning mechanism.

ABSTRACT

Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks. However, current algorithms fall short of offline backpropagation by either not being scalable or failing to learn long-range dependencies. Here we present a high-performance online learning algorithm that merely doubles the memory and computational requirements of a single inference pass. We achieve this by leveraging independent recurrent modules in multi-layer networks, an architectural motif that has recently been shown to be particularly powerful. Experiments on synthetic memory problems and on the challenging long-range arena benchmark suite reveal that our algorithm performs competitively, establishing a new standard for what can be achieved through online learning. This ability to learn long-range dependencies offers a new perspective on learning in the brain and opens a promising avenue in neuromorphic computing.

Motivation & Objective

• To address the challenge of online learning in recurrent networks where backpropagation-through-time (BPTT) is infeasible due to memory constraints.
• To enable exact online gradient computation in deep RNNs without relying on approximate or heuristic gradient estimation methods.
• To investigate whether architectural design—specifically independent recurrent modules—can simplify online credit assignment and make exact online learning tractable.
• To evaluate the method on synthetic memory tasks and the long-range arena benchmark, comparing performance to BPTT, truncated BPTT, and prior online learning methods.

Proposed method

• The method uses a multi-layer architecture composed of independent recurrent modules, where each recurrent unit operates in isolation, decoupling parameters across neurons within a layer.
• It applies exact complex differentiation to compute gradients in real time within each layer, enabling precise temporal credit assignment without storing full activation trajectories.
• Spatial backpropagation is used across layers to propagate error signals forward in time, while within-layer gradients are computed exactly using complex-valued state dynamics.
• The algorithm restricts recurrent connectivity to diagonal matrices and uses complex-valued hidden states, enabling efficient and exact online learning with minimal overhead.
• The approach is compatible with causal training by replacing time-pooling losses with per-step local losses and using layer normalization instead of batch normalization.
• It is implemented using deep linear state space models, including variants like LRU and HiPPO, which support high expressiveness with low-dimensional, independent recurrent units.

Experimental results

Research questions

• RQ1 Can exact online gradient computation be achieved in deep RNNs with only a small increase in memory and computation relative to inference?
• RQ2 Does the use of independent recurrent modules significantly improve online learning performance on long-range temporal tasks?
• RQ3 How does the proposed method compare to BPTT, truncated BPTT, and prior online learning approximations in terms of learning long-range dependencies?
• RQ4 Can the architecture of independent recurrent modules serve as a biologically plausible mechanism for online credit assignment in the brain?

Key findings

• The proposed online learning algorithm achieves performance on the long-range arena benchmark that is significantly closer to full BPTT than any prior online method, with a performance gap reduced by over 50% on average.
• On a synthetic copy task with sequence lengths up to 4,000 steps, the method successfully learns long-range dependencies, outperforming both spatial backpropagation and approximate real-time recurrent learning.
• The algorithm maintains strong performance even when using smaller models and causal training settings, demonstrating robustness and scalability.
• Replacing standard RNNs with linear RNNs or LRU layers in the CIFAR-100 experiment leads to improved online learning performance, confirming the effectiveness of independent recurrent modules.
• The method achieves exact gradient computation within a layer with only twice the memory and computational cost of a single inference pass, making it highly efficient for online learning.
• The results suggest that architectural modularity—specifically independent recurrent units—can drastically simplify online credit assignment, offering a plausible mechanism for learning in the brain and neuromorphic hardware.