[Paper Review] Generalizable Features From Unsupervised Learning

Mehdi Mirza, Aaron Courville+1 more|arXiv (Cornell University)|Dec 12, 2016

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes using unsupervised video prediction to learn generalizable features for improved zero-shot generalization in physical reasoning tasks. By training a generative model to predict future frames of block tower configurations, the model extracts features that significantly boost stability prediction accuracy on unseen configurations, outperforming supervised baselines and human performance on out-of-distribution test sets, especially for towers with more blocks than in training.

ABSTRACT

Humans learn a predictive model of the world and use this model to reason about future events and the consequences of actions. In contrast to most machine predictors, we exhibit an impressive ability to generalize to unseen scenarios and reason intelligently in these settings. One important aspect of this ability is physical intuition(Lake et al., 2016). In this work, we explore the potential of unsupervised learning to find features that promote better generalization to settings outside the supervised training distribution. Our task is predicting the stability of towers of square blocks. We demonstrate that an unsupervised model, trained to predict future frames of a video sequence of stable and unstable block configurations, can yield features that support extrapolating stability prediction to blocks configurations outside the training set distribution

Motivation & Objective

• To investigate whether unsupervised learning can produce features that enable better generalization to unseen physical configurations than supervised learning alone.
• To explore if a predictive model of future video frames—learned without labels—can improve zero-shot generalization in a block-stability prediction task.
• To evaluate whether the features learned from unsupervised video prediction transfer effectively to a downstream supervised task, particularly on out-of-distribution test scenarios.
• To compare the performance of models trained with generated data against supervised and human baselines, especially on configurations with more blocks than seen during training.
• To analyze the impact of data augmentation and model architecture (e.g., ConvDeconv vs. ConvLSTMDeconv) on generalization performance.

Proposed method

• Train a frame prediction model using an unsupervised objective to predict future video frames of block tower dynamics, using a physics engine to generate realistic sequences.
• Use the trained frame predictor to generate synthetic future frames, which are then used to augment the training data for a downstream stability prediction model.
• Train a supervised stability classifier (e.g., using AlexNet or a custom CNN) on real data augmented with generated future frames from the unsupervised model.
• Compare models trained with different data augmentation strategies: real data only, real + generated frames from ConvDeconv, and real + generated frames from ConvLSTMDeconv.
• Evaluate generalization performance on test sets with tower configurations (number of blocks) not seen during training, including both fewer and more blocks than in the training distribution.
• Incorporate the number of fallen blocks as a weak supervision signal in some experiments, though the primary task remains binary stability prediction.

Experimental results

Research questions

• RQ1 Can features learned via unsupervised video prediction improve generalization to out-of-distribution block tower configurations compared to purely supervised learning?
• RQ2 Does generating future frames through an unsupervised model lead to better zero-shot generalization on stability prediction, especially for configurations with more blocks than in training?
• RQ3 How does the quality of generated frames (e.g., sharpness, temporal coherence) affect downstream performance in stability prediction?
• RQ4 To what extent does data augmentation using generated frames improve generalization, particularly in low-data regimes?
• RQ5 Can the unsupervised model’s internal representations support better reasoning about physical stability than direct supervised learning on the same data?

Key findings

• The 4CD model (ConvDeconv-based generation) achieved 80.53% accuracy on test sets with 3 blocks when trained on 4-block towers—significantly outperforming the 52.5% accuracy of the 4S baseline without data augmentation.
• The 4CD model generalized better to configurations with more blocks than in training (e.g., 5 blocks) than to fewer blocks, with 89.1% accuracy on 5-block test sets, compared to 80.53% on 3-block tests.
• The 5CD model achieved 88.53% accuracy on 5-block test sets, outperforming the 5S baseline (67.23%) and approaching human performance (59%) on the same test set.
• Data augmentation using generated frames increased the 5CD model’s accuracy from 58.27% (no augmentation) to 74.50% on 4-block test sets, demonstrating a substantial generalization gain.
• Models using ConvDeconv for frame generation produced sharper, less noisy samples than ConvLSTMDeconv, correlating with better downstream performance (e.g., 92.5% vs. 91.20% accuracy for 4CD vs. 4CLD on 4-block test sets).
• Despite access to the number of fallen blocks as a weak signal, no significant improvement was observed, likely due to the highly unbalanced distribution of fallen block counts in the data.