[Paper Review] Disentangling Motion, Foreground and Background Features in Videos

Xunyu Lin, Víctor Campos+3 more|arXiv (Cornell University)|Jul 13, 2017

Advanced Vision and Imaging13 references3 citations

TL;DR

This paper proposes an unsupervised deep learning framework that disentangles motion, foreground, and background features in videos using a 3D convolutional autoencoder architecture. By training on frame reconstruction with motion-aware disentanglement, the model learns semantically meaningful features that significantly improve action recognition accuracy—achieving 62.5% on UCF-101 test set, a 10%+ gain over random initialization and autoencoder pretraining.

ABSTRACT

This paper introduces an unsupervised framework to extract semantically rich features for video representation. Inspired by how the human visual system groups objects based on motion cues, we propose a deep convolutional neural network that disentangles motion, foreground and background information. The proposed architecture consists of a 3D convolutional feature encoder for blocks of 16 frames, which is trained for reconstruction tasks over the first and last frames of the sequence. A preliminary supervised experiment was conducted to verify the feasibility of proposed method by training the model with a fraction of videos from the UCF-101 dataset taking as ground truth the bounding boxes around the activity regions. Qualitative results indicate that the network can successfully segment foreground and background in videos as well as update the foreground appearance based on disentangled motion features. The benefits of these learned features are shown in a discriminative classification task, where initializing the network with the proposed pretraining method outperforms both random initialization and autoencoder pretraining. Our model and source code are publicly available at https://imatge-upc.github.io/unsupervised-2017-cvprw/ .

Motivation & Objective

• To develop an unsupervised video representation learning framework that mimics human perceptual grouping using motion cues.
• To disentangle motion, foreground, and background features in video clips without relying on expensive human annotations.
• To improve discriminative action recognition performance through pretraining with disentangled features.
• To demonstrate that motion-based disentanglement leads to better generalization than random or autoencoder pretraining.

Proposed method

• A 3D convolutional neural network encoder processes 16-frame video clips and maps them to a shared bottleneck feature space.
• The bottleneck features are split into three distinct heads: one for motion, one for foreground appearance, and one for background appearance.
• Each head is trained to reconstruct its respective component (motion, foreground, background) from the first and last frames of the clip using a reconstruction loss.
• The architecture uses a multi-stream decoder setup, with separate decoders for motion, foreground, and background reconstruction.
• Ground truth segmentation masks are used during training, but the method is designed to eventually support unsupervised foreground segmentation via methods like uNLC.
• The model is fine-tuned for action recognition by replacing the decoders with a linear softmax layer on top of the disentangled features.

Experimental results

Research questions

• RQ1 Can motion cues alone enable unsupervised disentanglement of foreground and background in videos?
• RQ2 Does disentangling motion, foreground, and background lead to better video representation learning than standard autoencoding?
• RQ3 Can the proposed pretraining method improve action recognition performance with limited labeled data?
• RQ4 How does the model’s performance compare to random initialization and autoencoder pretraining on downstream tasks?
• RQ5 Can the model generalize to real-world video understanding tasks without human-annotated segmentation?

Key findings

• The model achieved 62.5% accuracy on the UCF-101 test set when fine-tuned after pretraining with disentangled features, outperforming random initialization (52.2%) and autoencoder pretraining (56.8%).
• The pretrained model reached 90% training accuracy after just one epoch, while the randomly initialized model required 130 epochs to reach the same performance.
• Qualitative results show that the network successfully segments moving objects as foreground and reconstructs motion features that capture both global transitions and local pose changes.
• The disentangled motion features enable accurate reconstruction of object shifts and gesture changes between frames, indicating effective modeling of temporal dynamics.
• The method demonstrates strong generalization, especially under data-scarce conditions, due to the inductive bias of motion-based disentanglement.
• The framework is publicly released with code and models available at https://imatge-upc.github.io/unsupervised-2017-cvprw/.