[Paper Review] Convolutional Tensor-Train LSTM for Spatio-temporal Learning

Jiahao Su, Wonmin Byeon+4 more|arXiv (Cornell University)|Feb 21, 2020

Tensor decomposition and applications59 references75 citations

TL;DR

This paper introduces a higher-order Convolutional Tensor-Train LSTM (Conv-TT-LSTM) for learning long-term spatio-temporal correlations in videos, using a convolutional tensor-train decomposition to compressedly model multiple past states and achieve state-of-the-art results on video prediction and early activity recognition.

ABSTRACT

Learning from spatio-temporal data has numerous applications such as human-behavior analysis, object tracking, video compression, and physics simulation.However, existing methods still perform poorly on challenging video tasks such as long-term forecasting. This is because these kinds of challenging tasks require learning long-term spatio-temporal correlations in the video sequence. In this paper, we propose a higher-order convolutional LSTM model that can efficiently learn these correlations, along with a succinct representations of the history. This is accomplished through a novel tensor train module that performs prediction by combining convolutional features across time. To make this feasible in terms of computation and memory requirements, we propose a novel convolutional tensor-train decomposition of the higher-order model. This decomposition reduces the model complexity by jointly approximating a sequence of convolutional kernels asa low-rank tensor-train factorization. As a result, our model outperforms existing approaches, but uses only a fraction of parameters, including the baseline models.Our results achieve state-of-the-art performance in a wide range of applications and datasets, including the multi-steps video prediction on the Moving-MNIST-2and KTH action datasets as well as early activity recognition on the Something-Something V2 dataset.

Motivation & Objective

• Motivate the need for modeling long-term spatio-temporal correlations in video data beyond first-order ConvLSTM.
• Develop a computationally efficient higher-order ConvLSTM using a novel convolutional tensor-train decomposition (CTTD).
• Preserve spatial structure while expanding temporal receptive fields with linear-time factorization in N time-steps.
• Demonstrate strong empirical performance on video prediction and early activity recognition across multiple datasets.
• Provide training strategies and architectural components that stabilize and improve higher-order spatio-temporal learning.

Proposed method

• Propose a higher-order ConvLSTM that aggregates information from multiple past hidden states via a function Φ(H(t−1),…,H(t−N)).
• Introduce Convolutional Tensor-Train Decomposition (CTTD) to factorize the sequence of convolutional kernels K(i) into a chain of smaller factors G(j), enabling linear growth in parameters with respect to the history order N.
• Use a preprocessing module to create overlapping, channel-reduced representations ˜H(i) from past hidden states to mitigate gradient issues and control dimensionality.
• Define the Conv-TT-LSTM update as [I(t); F(t); ˜C(t); O(t)] = σ(W ∗ X(t) + CTTD{G(1),…,G(N)} ∗ [˜H(1),…,˜H(N)]).
• Show that the CTT module computation can be performed in linear time O(N), satisfying efficiency and scalability constraints.
• Provide training strategies including learning rate schedules, scheduled sampling, curriculum learning, and gradient clipping to stabilize higher-order training.

Experimental results

Research questions

• RQ1 Can a higher-order ConvLSTM preserve spatial structure while incorporating longer temporal history without exponential growth in parameters?
• RQ2 Does Convolutional Tensor-Train Decomposition (CTTD) enable efficient, scalable modeling of long-range spatio-temporal dependencies?
• RQ3 Do preprocessing and training strategies improve optimization and generalization of higher-order spatio-temporal models?
• RQ4 How does Conv-TT-LSTM perform on video prediction and early activity recognition compared to ConvLSTM, 3D-CNN, and state-of-the-art predictors?

Key findings

• Conv-TT-LSTM consistently outperforms ConvLSTM baselines in video prediction and early activity recognition across evaluated datasets.
• On Something-Something v2, the model achieves an absolute 8% accuracy improvement over the state-of-the-art.
• For multi-step video prediction, Conv-TT-LSTM establishes new state-of-the-art results on Moving-MNIST-2 and KTH datasets.
• CTTD reduces parameter count and FLOPs while maintaining or improving predictive performance, enabling long-horizon spatio-temporal modeling.
• A principled preprocessing module and training strategies are essential for stabilizing and improving higher-order models.