[Paper Review] PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning

Yunbo Wang, Zhifeng Gao+3 more|arXiv (Cornell University)|Apr 17, 2018

Machine Learning and Data ClassificationComputer Science315 citations

TL;DR

PredRNN++ introduces a cascaded causal LSTM with dual memories and a Gradient Highway Unit to enable deeper-in-time spatiotemporal modeling while mitigating vanishing gradients, achieving state-of-the-art video prediction on synthetic and real datasets.

ABSTRACT

We present PredRNN++, an improved recurrent network for video predictive learning. In pursuit of a greater spatiotemporal modeling capability, our approach increases the transition depth between adjacent states by leveraging a novel recurrent unit, which is named Causal LSTM for re-organizing the spatial and temporal memories in a cascaded mechanism. However, there is still a dilemma in video predictive learning: increasingly deep-in-time models have been designed for capturing complex variations, while introducing more difficulties in the gradient back-propagation. To alleviate this undesirable effect, we propose a Gradient Highway architecture, which provides alternative shorter routes for gradient flows from outputs back to long-range inputs. This architecture works seamlessly with causal LSTMs, enabling PredRNN++ to capture short-term and long-term dependencies adaptively. We assess our model on both synthetic and real video datasets, showing its ability to ease the vanishing gradient problem and yield state-of-the-art prediction results even in a difficult objects occlusion scenario.

Motivation & Objective

• Motivate deeper-in-time modeling for short-term video dynamics in spatiotemporal predictive learning.
• Propose a cascaded dual-memory mechanism (causal LSTM) to increase recurrence depth from one time step to the next.
• Address vanishing gradients in deep recurrent structures with a Gradient Highway Unit (GHU).
• Demonstrate that combining causal LSTMs with GHU yields adaptive capture of short-term and long-term dependencies.
• Evaluate on synthetic and real video datasets to show state-of-the-art performance.

Proposed method

• Introduce causal LSTM with cascaded dual memories (temporal memory C_t^k and spatial memory M_t^k) to increase recurrence depth between time steps.
• Connect memories via cascaded gates to create a deeper transition path and larger receptive field for each predicted frame.
• Propose Gradient Highway Unit (GHU) to provide short-cut routes for gradient flow from outputs back to distant inputs, controlled by a Switch Gate S_t.
• Stack L causal LSTMs and insert a single GHU between the first and second LSTM layers to balance long-term and short-term information flows.
• Adopt 5-layer architecture: 4 causal LSTMs with channel sizes 128, 64, 64, 64 and a 128-channel GHU; use 5x5 convolutions.

Experimental results

Research questions

• RQ1 Can deeper-in-time recurrent architectures improve short-term video dynamics modeling without suffering from vanishing gradients?
• RQ2 Does a cascaded dual-memory (causal LSTM) structure better capture spatial-temporal dependencies than previous ST-LSTM or deep transition RNNs?
• RQ3 Can a Gradient Highway Unit provide effective long-range gradient propagation to support learning from distant frames?
• RQ4 How does PredRNN++ perform on synthetic occlusion-heavy data and real action videos compared to prior state-of-the-art models?
• RQ5 What is the impact of placing the GHU at different locations in the network?

Key findings

• PredRNN++ with causal LSTM and GHU achieves state-of-the-art results on Moving MNIST variants, outperforming several baselines including PredRNN and ST-LSTM variants.
• A cascaded, spatial-to-temporal causal LSTM variant improves SSIM and reduces MSE versus the original ST-LSTM.
• GHU consistently boosts performance across models, with the best gains when placed between the bottom two LSTM layers.
• Gradient analysis shows PredRNN++ maintains stronger, more informative gradients across time, mitigating the vanishing gradient problem during long-range prediction.
• On KTH action data, PredRNN++ achieves the highest PSNR and SSIM, indicating improved long-term dependency modeling and less blurring.