[Paper Review] Unitary Evolution Recurrent Neural Networks

Martín Arjovsky, Amar Shah+1 more|arXiv (Cornell University)|Nov 20, 2015

Neural Networks and Applications13 references222 citations

TL;DR

This paper proposes Unitary Evolution Recurrent Neural Networks (uRNNs), which use a novel parameterization of unitary weight matrices to stabilize gradients and enable learning of long-term dependencies. By composing structured unitary matrices in the complex domain, the model achieves efficient computation and state-of-the-art performance on tasks requiring long-context reasoning, outperforming LSTMs and orthogonally initialized RNNs.

ABSTRACT

Recurrent neural networks (RNNs) are notoriously difficult to train. When the eigenvalues of the hidden to hidden weight matrix deviate from absolute value 1, optimization becomes difficult due to the well studied issue of vanishing and exploding gradients, especially when trying to learn long-term dependencies. To circumvent this problem, we propose a new architecture that learns a unitary weight matrix, with eigenvalues of absolute value exactly 1. The challenge we address is that of parametrizing unitary matrices in a way that does not require expensive computations (such as eigendecomposition) after each weight update. We construct an expressive unitary weight matrix by composing several structured matrices that act as building blocks with parameters to be learned. Optimization with this parameterization becomes feasible only when considering hidden states in the complex domain. We demonstrate the potential of this architecture by achieving state of the art results in several hard tasks involving very long-term dependencies.

Motivation & Objective

• To address the vanishing and exploding gradient problems in recurrent neural networks (RNNs) that hinder learning of long-term dependencies.
• To develop an efficient parameterization of unitary matrices that avoids costly eigendecompositions after each weight update.
• To enable stable gradient flow through long sequences by constraining the hidden-to-hidden weight matrix to be unitary (eigenvalues of absolute value 1).
• To explore the benefits of complex-valued recurrent networks for improved optimization and representation capacity.
• To demonstrate that uRNNs can learn long-term dependencies more effectively than LSTMs and orthogonally initialized RNNs.

Proposed method

• Proposes a parameterization of unitary matrices as a composition of structured, learnable matrices (e.g., diagonal, permutation, and rotation matrices) to ensure unitarity without eigendecomposition.
• Uses complex-valued hidden states and parameters to enable effective optimization of unitary matrices, which is difficult in the real domain.
• Employs a matrix factorization approach where each component matrix is parameterized with O(n log n) computation and O(n) memory, unlike general matrices requiring O(n²) operations.
• Applies the chain rule to backpropagate gradients, leveraging the fact that the inverse of a unitary matrix is its conjugate transpose, enabling efficient backward passes.
• Uses real-valued backpropagation in Theano by representing complex matrices as real 2n×2n blocks, enabling standard optimization.
• Implements a differentiable, parameterized unitary matrix construction that maintains unitarity during training via constrained parameter updates.

Experimental results

Research questions

• RQ1 Can a parameterization of unitary matrices be constructed that avoids expensive eigendecompositions while preserving unitarity during training?
• RQ2 Does using unitary weight matrices in RNNs significantly improve gradient flow through long sequences compared to standard RNNs and LSTMs?
• RQ3 Can complex-valued recurrent networks outperform real-valued counterparts in learning long-term dependencies?
• RQ4 To what extent does the uRNN avoid hidden state saturation compared to LSTMs when processing very long sequences?
• RQ5 Can the uRNN achieve state-of-the-art performance on tasks requiring reasoning over very long sequences?

Key findings

• The uRNN maintains stable gradient norms over long sequences, with minimal decay, while LSTMs and standard RNNs exhibit exponentially vanishing gradients.
• After 100 training iterations, the uRNN still propagates gradients with significantly less decay than other models, indicating superior gradient flow.
• The uRNN does not suffer from hidden state saturation: its hidden state norms grow steadily over time, unlike LSTMs, which plateau after ~500 time steps.
• In the adding problem with T=750, the LSTM showed a Pearson correlation of ρ=0.991 between its output and the first input, indicating it stored only the first value and failed to incorporate the second.
• The uRNN achieved state-of-the-art performance on hard tasks involving long-term dependencies, outperforming LSTMs and orthogonally initialized RNNs.
• The model enables potential memory reduction by allowing re-computation of hidden states via the inverse of the unitary matrix, eliminating the need to store all states during backpropagation.