[Paper Review] Learning Long-Term Dependencies in Irregularly-Sampled Time Series

Mathias Lechner, Ramin Hasani|arXiv (Cornell University)|Jun 8, 2020

Model Reduction and Neural Networks49 references56 citations

TL;DR

The paper introduces ODE-LSTMs, a continuous-time extension of LSTMs that decouples memory from the time-continuous state to reliably learn long-term dependencies in irregularly-sampled data, addressing vanishing/exploding gradients in ODE-RNNs and achieving superior performance on various tasks.

ABSTRACT

Recurrent neural networks (RNNs) with continuous-time hidden states are a natural fit for modeling irregularly-sampled time series. These models, however, face difficulties when the input data possess long-term dependencies. We prove that similar to standard RNNs, the underlying reason for this issue is the vanishing or exploding of the gradient during training. This phenomenon is expressed by the ordinary differential equation (ODE) representation of the hidden state, regardless of the ODE solver's choice. We provide a solution by designing a new algorithm based on the long short-term memory (LSTM) that separates its memory from its time-continuous state. This way, we encode a continuous-time dynamical flow within the RNN, allowing it to respond to inputs arriving at arbitrary time-lags while ensuring a constant error propagation through the memory path. We call these RNN models ODE-LSTMs. We experimentally show that ODE-LSTMs outperform advanced RNN-based counterparts on non-uniformly sampled data with long-term dependencies. All code and data is available at https://github.com/mlech26l/ode-lstms.

Motivation & Objective

• Motivation to model irregularly-sampled time series with long-term dependencies.
• Identify why ODE-RNNs face vanishing/exploding gradients during training.
• Propose a memory-augmented continuous-time RNN that preserves gradient flow.
• Demonstrate empirical superiority of the proposed model across synthetic and real-world tasks.

Proposed method

• Propose ODE-LSTM by keeping the LSTM memory cell and gating while solving a continuous-time output dynamics with an ODE-RNN pathway.
• Prove theoretically that ODE-RNNs suffer from vanishing/exploding gradients under common discretizations and adjoint training.
• Integrate a time-continuous flow into the LSTM output computation to enable responses to arbitrary time-lags while maintaining stable gradient propagation.
• Compare ODE-LSTM to a broad set of continuous-time RNN baselines (e.g., ODE-RNN, CT-RNN, GRU-ODE, CT-LSTM, GRU-D) on synthetic and real datasets.
• Provide empirical results across bit-stream XOR-like tasks, activity recognition, irregular sequential MNIST, and Walker2d kinematics.

Experimental results

Research questions

• RQ1 Do ODE-RNNs suffer from vanishing/exploding gradients when learning long-term dependencies on irregularly-sampled data?
• RQ2 Can decoupling memory from the time-continuous state via an LSTM-like memory in a continuous-time RNN enable stable gradient flow?
• RQ3 Do ODE-LSTMs outperform existing continuous-time RNN variants on synthetic and real irregular time-series benchmarks?
• RQ4 How does the proposed model perform on tasks requiring long-term dependency learning with non-uniform sampling?

Key findings

• ODE-RNNs and related continuous-time RNNs exhibit vanishing or exploding gradients, hindering long-term dependency learning.
• ODE-LSTMs maintain near-constant gradient flow through the memory path, enabling learning of long-term dependencies in irregularly-sampled data.
• Across synthetic and real-world tasks, ODE-LSTMs consistently outperform advanced continuous-time RNN variants.
• On tasks like bit-stream XOR, irregular MNIST, and Walker2d dynamics, ODE-LSTMs achieve superior performance.
• The architecture effectively handles irregular sampling without decaying memory, unlike several decay-based baselines.