[Paper Review] Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

S. M. Ali Eslami, Nicolas Heess+5 more|arXiv (Cornell University)|Mar 28, 2016

Generative Adversarial Networks and Image Synthesis32 references248 citations

TL;DR

AIR introduces an amortized, recurrent attention-based inference framework for unsupervised scene parsing with variable numbers of objects, enabling fast, interpretable 2D and 3D scene understanding through end-to-end learning.

ABSTRACT

We present a framework for efficient inference in structured image models that explicitly reason about objects. We achieve this by performing probabilistic inference using a recurrent neural network that attends to scene elements and processes them one at a time. Crucially, the model itself learns to choose the appropriate number of inference steps. We use this scheme to learn to perform inference in partially specified 2D models (variable-sized variational auto-encoders) and fully specified 3D models (probabilistic renderers). We show that such models learn to identify multiple objects - counting, locating and classifying the elements of a scene - without any supervision, e.g., decomposing 3D images with various numbers of objects in a single forward pass of a neural network. We further show that the networks produce accurate inferences when compared to supervised counterparts, and that their structure leads to improved generalization.

Motivation & Objective

• Motivate and formalize a Bayesian, generative-model approach to scene understanding that decomposes scenes into objects.
• Develop an efficient, amortized inference mechanism that can handle variable object counts and spatial attention.
• Show that structured AIR models yield interpretable representations and improved generalization over unstructured VAEs.
• Demonstrate AIR's applicability to both partially specified 2D models and fully specified 3D renderers.
• Highlight the speed and robustness of inference compared with traditional optimization or supervised approaches.

Proposed method

• Formulate p_theta(x) as a sum over an (unknown) number of objects with latent per-object variables z^i and a presence variable z_pres^i.
• Use an amortized variational posterior q_phi(z, n | x) implemented as an iterative recurrent network that attends to one object per time step and can terminate when z_pres indicates no more objects.
• Train end-to-end by maximizing the evidence lower bound with respect to theta and phi, employing reparameterization for continuous variables and likelihood-ratio estimators for discrete ones.
• In the 2D setting, model each object with a learned shape code z_what and a pose z_where; in the 3D setting, model objects with identity and pose variables and render via a probabilistic renderer.
• Incorporate spatial transformers and attention to extract object-focused patches for progressive reconstruction and inference.
• Demonstrate that AIR can decompose multi-object scenes in an unsupervised fashion and perform fast 3D scene inference with counts, identities, and poses.

Experimental results

Research questions

• RQ1 Can a recurrent, attention-based inference network accurately infer the number of objects and their attributes in a scene without supervision?
• RQ2 How does incorporating structured priors and variable-length inference affect generalization and reconstruction quality relative to unstructured generative models?
• RQ3 Can AIR perform rapid, amortized inference for both 2D partially specified models and fully specified 3D renderers?
• RQ4 Does the learned representation support useful downstream tasks such as counting, localization, and classification with limited labeled data?
• RQ5 How does AIR compare to supervised or non-amortized inference methods in speed, robustness, and avoidance of local minima?

Key findings

• AIR learns to count, locate, and classify scene elements in an unsupervised manner.
• The iterative, attended inference network can terminate early when no objects remain, enabling variable-length computation.
• AIR provides fast, forward-pass inferences that decompose scenes with multiple objects and occlusions, outperforming some non-structured baselines in generalization.
• In 2D experiments, AIR shows strong generalization to unseen counts and configurations, including comparisons with DRAW and DAIR.
• In 3D experiments, AIR infers counts, identities, and poses from rendered and real table-top scenes, often more robustly than fully supervised approaches and with reduced optimization difficulties.