[Paper Review] Binge Watching: Scaling Affordance Learning from Sitcoms

Xiaolong Wang, Rohit Girdhar+1 more|arXiv (Cornell University)|Apr 9, 2018

Robot Manipulation and Learning36 references3 citations

TL;DR

This paper introduces a large-scale dataset of 28,882 affordance poses extracted from 100 million frames of seven sitcoms, enabling scalable learning of human-object interaction. It proposes a two-stage method: first classifying likely pose classes at a scene location, then using a Variational Autoencoder (VAE) to generate diverse, realistic scale and deformation variations, significantly improving generalization over baseline models.

ABSTRACT

In recent years, there has been a renewed interest in jointly modeling perception and action. At the core of this investigation is the idea of modeling affordances(Affordances are opportunities of interaction in the scene. In other words, it represents what actions can the object be used for). However, when it comes to predicting affordances, even the state of the art approaches still do not use any ConvNets. Why is that? Unlike semantic or 3D tasks, there still does not exist any large-scale dataset for affordances. In this paper, we tackle the challenge of creating one of the biggest dataset for learning affordances. We use seven sitcoms to extract a diverse set of scenes and how actors interact with different objects in the scenes. Our dataset consists of more than 10K scenes and 28K ways humans can interact with these 10K images. We also propose a two-step approach to predict affordances in a new scene. In the first step, given a location in the scene we classify which of the 30 pose classes is the likely affordance pose. Given the pose class and the scene, we then use a Variational Autoencoder (VAE) to extract the scale and deformation of the pose. The VAE allows us to sample the distribution of possible poses at test time. Finally, we show the importance of large-scale data in learning a generalizable and robust model of affordances.

Motivation & Objective

• To address the lack of large-scale, diverse datasets for learning visual affordances in robotics and computer vision.
• To develop a scalable, data-efficient method for predicting human interaction poses in scenes without relying on physical simulations or manual labeling.
• To improve the generalization and robustness of affordance prediction by leveraging diverse, real-world human interaction patterns from TV shows.
• To demonstrate that large-scale, weakly supervised data from scripted scenes can effectively train models for functional understanding of environments.

Proposed method

• Extract 10,000+ scenes and 28,882 human-object interaction poses from 100M frames of seven sitcoms using automatic frame processing and scene registration.
• Use a 30-class pose classification network to predict the most likely affordance pose type at a given scene location.
• Train a Variational Autoencoder (VAE) to model the distribution of scale and deformation for each predicted pose class, enabling stochastic sampling at test time.
• Apply VAE decoding with random latent noise vectors to generate multiple plausible, diverse poses for the same scene and location.
• Use a heatmap-based baseline for comparison, where predicted heatmaps are matched to cluster centers for pose classification.
• Evaluate both classification and pose realism using human evaluation and precision-recall curves on positive and negative test samples.

Experimental results

Research questions

• RQ1 Can large-scale, weakly supervised data from scripted TV shows be effectively used to train models for visual affordance prediction?
• RQ2 Does increasing the scale of training data lead to improved generalization and robustness in affordance prediction?
• RQ3 Can a VAE-based approach generate diverse, realistic pose variations that match human intuition better than deterministic or averaged outputs?
• RQ4 How does the proposed two-stage method (classification + VAE refinement) compare to baseline heatmap-based approaches in pose classification accuracy?

Key findings

• Training with 25K data points achieved a top-5 classification accuracy of 50.9%, significantly outperforming the heatmap baseline (47.3%).
• The model’s predicted poses were judged by humans as more realistic than ground truth in 46% of cases, approaching human-level discrimination performance.
• The VAE-based method successfully generated multiple plausible pose variations for the same scene and location, demonstrating its ability to model pose distribution uncertainty.
• The precision-recall curve showed that the model trained on 25K data points achieved the best performance in classifying whether a given pose is plausible in a scene.
• The results indicate that large-scale, real-world interaction data from sitcoms can effectively train models that generalize better than those trained on smaller or synthetic datasets.