[Paper Review] Approximating Real-Time Recurrent Learning with Random Kronecker Factors

Asier Mujika, Florian Meier+1 more|arXiv (Cornell University)|May 28, 2018

Domain Adaptation and Few-Shot Learning19 citations

TL;DR

This paper proposes KF-RTRL, a memory-efficient, unbiased online learning algorithm that uses Kronecker product decomposition to approximate real-time recurrent learning (RTRL) gradients. It achieves stable, low-noise gradients and matches TBPTT performance on long-sequence tasks, offering a practical alternative to truncated backpropagation.

ABSTRACT

Despite all the impressive advances of recurrent neural networks, sequential data is still in need of better modelling. Truncated backpropagation through time (TBPTT), the learning algorithm most widely used in practice, suffers from the truncation bias, which drastically limits its ability to learn long-term dependencies.The Real Time Recurrent Learning algorithm (RTRL) addresses this issue, but its high computational requirements make it infeasible in practice. The Unbiased Online Recurrent Optimization algorithm (UORO) approximates RTRL with a smaller runtime and memory cost, but with the disadvantage of obtaining noisy gradients that also limit its practical applicability. In this paper we propose the Kronecker Factored RTRL (KF-RTRL) algorithm that uses a Kronecker product decomposition to approximate the gradients for a large class of RNNs. We show that KF-RTRL is an unbiased and memory efficient online learning algorithm. Our theoretical analysis shows that, under reasonable assumptions, the noise introduced by our algorithm is not only stable over time but also asymptotically much smaller than the one of the UORO algorithm. We also confirm these theoretical results experimentally. Further, we show empirically that the KF-RTRL algorithm captures long-term dependencies and almost matches the performance of TBPTT on real world tasks by training Recurrent Highway Networks on a synthetic string memorization task and on the Penn TreeBank task, respectively. These results indicate that RTRL based approaches might be a promising future alternative to TBPTT.

Motivation & Objective

• Address the high computational cost and memory requirements of exact RTRL, which limit its practical use despite solving truncation bias.
• Overcome the noisy gradient problem of UORO, an existing RTRL approximation, which hinders stable training and performance.
• Develop a memory-efficient, online learning algorithm that maintains gradient unbiasedness while reducing noise for better generalization.
• Enable effective learning of long-term dependencies in recurrent networks without relying on truncated backpropagation through time (TBPTT).

Proposed method

• Apply Kronecker product decomposition to approximate the Fisher information matrix in RTRL, reducing gradient computation to low-rank updates.
• Use structured factorization to maintain the full gradient computation path while drastically reducing storage and computational complexity.
• Maintain online learning capability by updating the Kronecker factors incrementally with each new time step.
• Ensure gradient unbiasedness by preserving the exact RTRL update direction through the factorized approximation.
• Control gradient noise by constraining the approximation error via the Kronecker structure, leading to asymptotically smaller noise than UORO.
• Integrate the method into Recurrent Highway Networks for empirical evaluation on sequence modeling benchmarks.

Experimental results

Research questions

• RQ1 Can Kronecker factorization be used to create a memory-efficient and unbiased approximation of RTRL gradients?
• RQ2 Does the proposed KF-RTRL method reduce gradient noise compared to UORO, especially over long training sequences?
• RQ3 Can KF-RTRL effectively learn long-term dependencies in recurrent networks, matching the performance of TBPTT?
• RQ4 How does KF-RTRL perform on real-world sequence modeling tasks such as language modeling and synthetic memorization tasks?

Key findings

• KF-RTRL achieves unbiased gradient estimation, maintaining the theoretical correctness of RTRL while significantly reducing computational cost.
• The gradient noise in KF-RTRL is asymptotically smaller and more stable over time than in UORO, as shown by theoretical analysis.
• On a synthetic string memorization task, KF-RTRL successfully captures long-term dependencies, demonstrating its ability to learn from extended sequences.
• On the Penn TreeBank language modeling task, KF-RTRL achieves performance nearly matching that of TBPTT, indicating strong practical viability.
• KF-RTRL reduces memory and runtime requirements compared to exact RTRL, making online training feasible for large-scale RNNs.
• Empirical results confirm that the theoretical noise reduction translates into more stable and effective training compared to UORO.