[Paper Review] Early Visual Concept Learning with Unsupervised Deep Learning

Irina Higgins, Löıc Matthey+6 more|arXiv (Cornell University)|Jun 17, 2016

Generative Adversarial Networks and Image Synthesis28 references127 citations

TL;DR

The paper shows that a variational autoencoder (VAE) with neuroscience-inspired constraints learns disentangled continuous visual factors from raw images, enabling zero-shot inference and emergent concepts like objectness without supervision.

ABSTRACT

Automated discovery of early visual concepts from raw image data is a major open challenge in AI research. Addressing this problem, we propose an unsupervised approach for learning disentangled representations of the underlying factors of variation. We draw inspiration from neuroscience, and show how this can be achieved in an unsupervised generative model by applying the same learning pressures as have been suggested to act in the ventral visual stream in the brain. By enforcing redundancy reduction, encouraging statistical independence, and exposure to data with transform continuities analogous to those to which human infants are exposed, we obtain a variational autoencoder (VAE) framework capable of learning disentangled factors. Our approach makes few assumptions and works well across a wide variety of datasets. Furthermore, our solution has useful emergent properties, such as zero-shot inference and an intuitive understanding of "objectness".

Motivation & Objective

• Demonstrate that unsupervised deep generative models can learn disentangled representations of continuous visual factors.
• Incorporate neuroscience-inspired pressures (data continuity, redundancy reduction, independence) into a VAE framework.
• Quantitatively evaluate disentanglement and show emergent properties and zero-shot inference.
• Show robustness across architectures, datasets, and noise settings.

Proposed method

• Formulate a VAE with a prior over latent factors and a beta-regularized objective to enforce redundancy reduction and independence (L = E_q[log p(x|z)] - beta * KL(q(z|x)||p(z))).
• Use an isotropic Gaussian prior to induce independence among latent factors.
• Train on datasets with continuous transformations to encourage manifold learning and disentanglement.
• Quantify disentanglement with a factor-change classifier predicting which generative factor caused a frame transition.
• Evaluate zero-shot generalization by testing on unseen factor combinations and new object identities.
• Demonstrate emergent concepts such as objectness by testing reasoning on novel objects.

Experimental results

Research questions

• RQ1 Can unsupervised deep generative models learn disentangled factors of visual variation without prior knowledge of the factors?
• RQ2 How do data continuity, redundancy reduction, and independence pressures affect disentanglement in VAEs?
• RQ3 Do disentangled representations enable zero-shot inference and generalization to novel objects?
• RQ4 What are the emergent properties of disentangled VAEs in terms of objectness and transfer to new tasks?

Key findings

• Disentangled VAEs learn latent units corresponding to distinct continuous factors (position, scale, rotation) given appropriate learning pressure.
• Reducing data continuity harms disentanglement, while higher beta values up to an optimum improve disentanglement.
• Larger latent spaces require stronger normalized beta, with an inverted U relationship between beta and disentanglement.
• Disentangled representations enable zero-shot reasoning about unseen factor combinations and novel objects.
• Disentangled VAEs allocate independent latent factors consistent with the statistics of the data, improving generalization to new combinations and tasks.
• Reconstruction quality alone is not a reliable indicator of disentanglement; disentangled models can produce blurrier reconstructions.