[Paper Review] Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics

Yunbo Wang, Jianjin Zhang+4 more|arXiv (Cornell University)|Nov 19, 2018

Meteorological Phenomena and Simulations38 references21 citations

TL;DR

This paper proposes Memory In Memory (MIM), a novel recurrent neural network architecture that models higher-order non-stationarity in spatiotemporal dynamics by leveraging differential signals between consecutive hidden states through two cascaded, self-renewed memory modules. MIM achieves state-of-the-art performance on four diverse spatiotemporal prediction tasks, including radar echo forecasting, traffic flow prediction, and human pose video generation, outperforming prior models like PredRNN and FRNN in both quantitative metrics and visual quality.

ABSTRACT

Natural spatiotemporal processes can be highly non-stationary in many ways, e.g. the low-level non-stationarity such as spatial correlations or temporal dependencies of local pixel values; and the high-level variations such as the accumulation, deformation or dissipation of radar echoes in precipitation forecasting. From Cramer's Decomposition, any non-stationary process can be decomposed into deterministic, time-variant polynomials, plus a zero-mean stochastic term. By applying differencing operations appropriately, we may turn time-variant polynomials into a constant, making the deterministic component predictable. However, most previous recurrent neural networks for spatiotemporal prediction do not use the differential signals effectively, and their relatively simple state transition functions prevent them from learning too complicated variations in spacetime. We propose the Memory In Memory (MIM) networks and corresponding recurrent blocks for this purpose. The MIM blocks exploit the differential signals between adjacent recurrent states to model the non-stationary and approximately stationary properties in spatiotemporal dynamics with two cascaded, self-renewed memory modules. By stacking multiple MIM blocks, we could potentially handle higher-order non-stationarity. The MIM networks achieve the state-of-the-art results on four spatiotemporal prediction tasks across both synthetic and real-world datasets. We believe that the general idea of this work can be potentially applied to other time-series forecasting tasks.

Motivation & Objective

• To address the challenge of modeling complex, higher-order non-stationarity in spatiotemporal dynamics, especially in real-world processes like precipitation forecasting and traffic flow.
• To overcome the limitations of existing RNNs—particularly simple forget gates in models like PredRNN—that fail to capture evolving trends and result in blurry, linear extrapolation-based predictions.
• To develop a memory architecture that effectively leverages differencing operations on hidden states to gradually stationarize non-stationary processes, improving long-term predictability.
• To enable more accurate and robust spatiotemporal forecasting by explicitly modeling both non-stationary and approximately stationary components through a hierarchical memory structure.

Proposed method

• The MIM block replaces the forget gate in standard LSTMs with two cascaded, self-renewed long short-term memory modules to model higher-order non-stationarity and approximate stationarity in spatiotemporal dynamics.
• The network uses differential signals between adjacent recurrent hidden states as input to the inner memory modules, enabling the model to capture time-variant trends without over-differencing the original input signals.
• The architecture applies differencing at the memory transition level rather than on raw inputs, preserving information while reducing the order of non-stationary polynomials in the dynamics.
• Multiple MIM blocks are stacked to allow hierarchical modeling of increasingly complex non-stationary behaviors, progressively transforming non-stationary processes into more predictable, stationary forms.
• The model maintains a standard LSTM cell for long-term memory while introducing two auxiliary recurrent modules to handle the derivative-like evolution of hidden states.
• The design is inspired by ARIMA time series models, where differencing transforms non-stationary processes into stationary ones, but adapts this idea into a deep learning framework with internal memory recursion.

Experimental results

Research questions

• RQ1 Can a recurrent neural network architecture effectively model higher-order non-stationarity in spatiotemporal sequences by leveraging differential signals between hidden states?
• RQ2 How does the hierarchical memory structure in MIM improve the modeling of complex, time-variant trends such as echo accumulation and dissipation in radar data?
• RQ3 To what extent does replacing the forget gate with cascaded memory modules enhance predictive performance compared to standard RNNs and LSTMs on non-stationary spatiotemporal tasks?
• RQ4 Does the MIM architecture reduce the reliance on linear extrapolation in video prediction, thereby mitigating the blurry effect common in prior models?
• RQ5 Can the MIM framework generalize across diverse spatiotemporal prediction tasks, including synthetic, traffic, weather, and human motion datasets?

Key findings

• On the radar echo forecasting task, MIM achieved the lowest MSE (27.8) and highest CSI scores (0.429 at CSI-30, 0.399 at CSI-40, 0.317 at CSI-50), outperforming FRNN, PredRNN, and Causal LSTM across all metrics.
• In the Human3.6M human pose video prediction task, MIM achieved an SSIM of 0.790, MSE of 429.9, and MAE of 1782.8, surpassing FRNN and PredRNN in both quantitative metrics and visual fidelity.
• On the synthetic flying digits dataset, MIM demonstrated superior generalization and long-horizon prediction accuracy, effectively capturing complex motion patterns.
• The frame-wise evaluation showed that MIM maintained the best performance throughout the 10-frame forecasting horizon, with consistently lower MSE and higher CSI values than competing models.
• The analysis revealed that over 80% of forget gates in PredRNN were saturated, indicating ineffective non-stationarity modeling, whereas MIM’s dual-memory mechanism effectively captured evolving trends.
• Visual comparisons confirmed that MIM accurately predicted the movement, deformation, and dissipation of radar echoes, while other models failed to track complex spatial-temporal dynamics.