[Paper Review] Spatio-temporal video autoencoder with differentiable memory

Viorica Pătrăucean, Ankur Handa+1 more|arXiv (Cornell University)|Nov 19, 2015

Advanced Vision and Imaging31 references235 citations

TL;DR

This paper proposes a spatio-temporal video autoencoder with a differentiable visual short-term memory using convolutional LSTM cells to learn motion representations unsupervised. By integrating an optical flow predictor and image sampler as a feedback loop, the model reconstructs future frames end-to-end, achieving state-of-the-art performance in weakly-supervised video semantic segmentation with minimal supervision.

ABSTRACT

We describe a new spatio-temporal video autoencoder, based on a classic spatial image autoencoder and a novel nested temporal autoencoder. The temporal encoder is represented by a differentiable visual memory composed of convolutional long short-term memory (LSTM) cells that integrate changes over time. Here we target motion changes and use as temporal decoder a robust optical flow prediction module together with an image sampler serving as built-in feedback loop. The architecture is end-to-end differentiable. At each time step, the system receives as input a video frame, predicts the optical flow based on the current observation and the LSTM memory state as a dense transformation map, and applies it to the current frame to generate the next frame. By minimising the reconstruction error between the predicted next frame and the corresponding ground truth next frame, we train the whole system to extract features useful for motion estimation without any supervision effort. We present one direct application of the proposed framework in weakly-supervised semantic segmentation of videos through label propagation using optical flow.

Motivation & Objective

• Address the lack of labelled video data for training deep video models by enabling unsupervised learning of motion representations.
• Overcome the limitations of standard convolutional networks in exploiting temporal redundancy in video sequences.
• Develop a biologically inspired visual short-term memory module using convolutional LSTMs to model dynamic visual changes.
• Enable end-to-end training of a video autoencoder that jointly learns motion estimation and reconstruction without supervision.
• Demonstrate the utility of the learned motion features in weakly-supervised semantic segmentation via label propagation.

Proposed method

• Propose a spatio-temporal video autoencoder architecture combining a standard convolutional image autoencoder with a nested temporal autoencoder using convolutional LSTM cells as a differentiable visual short-term memory.
• Use the convolutional LSTM to encode temporal context by integrating visual changes over time, modeling motion dynamics in a differentiable manner.
• Implement a temporal decoder based on a robust optical flow prediction module and an image sampler that applies the predicted flow to the current frame to generate the next frame.
• Introduce a built-in feedback loop where the predicted next frame is compared to the ground truth, enabling end-to-end backpropagation through time.
• Train the entire system by minimizing the reconstruction loss between predicted and ground truth frames, enabling unsupervised feature learning for motion.
• Adapt the architecture for weakly-supervised semantic segmentation by propagating labels through optical flow, using the flow as a geometric prior.

Experimental results

Research questions

• RQ1 Can a differentiable visual short-term memory module based on convolutional LSTMs improve unsupervised motion representation learning in videos?
• RQ2 How effective is the end-to-end training scheme with a feedback loop of predicted vs. ground truth frames for learning motion features?
• RQ3 To what extent can the learned motion features improve weakly-supervised semantic segmentation of videos?
• RQ4 Does the integration of optical flow estimation and frame prediction enhance segmentation quality compared to standard models?
• RQ5 How does the architecture compare to standard autoencoders and fully-connected LSTM-based models in terms of performance and parameter efficiency?

Key findings

• The proposed spatio-temporal autoencoder with convolutional LSTM memory outperforms standard autoencoders and fully-connected LSTM baselines in motion representation learning with fewer parameters.
• The model achieves a global average segmentation accuracy of 76.9% on the Camvid dataset using only frame-level labels, significantly improving over baseline SegNet’s 75.3%.
• The SegNet-flow model, which integrates optical flow constraints, shows substantial improvements in smoothing segmentation results, especially for large object classes.
• Despite the improvement, performance degrades for small and thin structures, primarily due to the limited training set size and downsampled flow estimation.
• The feedback loop enables effective end-to-end training and allows for easy experimentation on memory module design and resolution.
• The architecture demonstrates potential for modeling visual illusions and ambiguous motion percepts, suggesting biological plausibility in memory dynamics.