[Paper Review] Learning from Videos with Deep Convolutional LSTM Networks

Logan Courtney, R.S. Sreenivas|arXiv (Cornell University)|Apr 9, 2019

Human Pose and Action Recognition86 references4 citations

TL;DR

This paper proposes a deep convolutional LSTM network that jointly learns spatial and temporal features from video data, using 2D convolutions within LSTM units to capture spatiotemporal patterns at multiple scales. The model achieves state-of-the-art performance on the LRW lipreading dataset (85.2% accuracy) when pre-trained on LRS2, demonstrating its ability to automatically identify relevant spatiotemporal scales without architectural prior design.

ABSTRACT

This paper explores the use of convolution LSTMs to simultaneously learn spatial- and temporal-information in videos. A deep network of convolutional LSTMs allows the model to access the entire range of temporal information at all spatial scales of the data. We describe our experiments involving convolution LSTMs for lipreading that demonstrate the model is capable of selectively choosing which spatiotemporal scales are most relevant for a particular dataset. The proposed deep architecture also holds promise in other applications where spatiotemporal features play a vital role without having to specifically cater the design of the network for the particular spatiotemporal features existent within the problem. For the Lip Reading in the Wild (LRW) dataset, our model slightly outperforms the previous state of the art (83.4% vs. 83.0%) and sets the new state of the art at 85.2% when the model is pretrained on the Lip Reading Sentences (LRS2) dataset.

Motivation & Objective

• To develop a deep neural network architecture capable of jointly modeling spatial and temporal features in video sequences.
• To investigate whether convolutional LSTMs can automatically learn relevant spatiotemporal scales without prior architectural bias for specific tasks.
• To improve lipreading performance by leveraging both short-term dynamics and long-term context through a unified spatiotemporal learning framework.
• To provide insights into which spatiotemporal scales are most critical for video understanding tasks like lipreading.

Proposed method

• The model employs a deep stack of convolutional LSTM layers, where 2D convolutions are integrated within the LSTM cell to process spatial features at each time step.
• The architecture combines 2D convolutions with convolutional LSTMs, enabling the network to learn hierarchical spatiotemporal representations across multiple spatial scales.
• A sensitivity analysis is performed by periodically resetting the internal states of convolutional LSTM layers at different intervals (T = 1, 3, 5, 10, 15 frames) to evaluate the importance of temporal context at each spatial resolution.
• The network is trained end-to-end on video sequences, with feature maps processed at multiple scales (s/2, s/4, s/8, s/16), where s denotes the original spatial size.
• A residual connection design is applied to stabilize training and improve performance, similar to ResNet in 2D and 3D convolutions.
• The model is fine-tuned and evaluated on the LRW dataset, with ablation studies on state reset intervals to assess temporal dependency importance.

Experimental results

Research questions

• RQ1 Can a deep architecture built primarily from convolutional LSTMs outperform existing models in video understanding tasks like lipreading?
• RQ2 Which spatiotemporal scales (spatial resolution and temporal context length) are most critical for accurate lipreading?
• RQ3 Does the use of convolutional LSTMs enable automatic discovery of relevant spatiotemporal features without prior architectural design?
• RQ4 How does the performance of the convolutional LSTM model compare to hybrid architectures combining 3D convolutions and bidirectional LSTMs?
• RQ5 To what extent do hidden-to-hidden connections in convolutional LSTMs contribute to performance at different spatial scales?

Key findings

• The model achieves a new state-of-the-art accuracy of 85.2% on the LRW dataset when pre-trained on the LRS2 dataset, surpassing the previous SOTA of 83.0%.
• The model slightly outperforms the prior SOTA (83.4%) on the original LRW dataset without pretraining, demonstrating strong generalization.
• Performance drops significantly at the s/16 scale when temporal context is removed, indicating that long-term temporal dependencies are critical at the coarsest spatial level.
• The s/8 scale shows strong performance only when temporal context is preserved over ~10 frames, suggesting it captures medium-term sequences.
• The s/2 scale is most sensitive to state resets every frame, confirming that fine-grained, short-term dynamics are essential for lipreading at high spatial resolution.
• The s/4 scale maintains high performance regardless of state reset frequency, indicating that hidden-to-hidden connections at this scale contribute little beyond what is already captured by the convolutional operations.