[Paper Review] SCAN: Learning Hierarchical Compositional Visual Concepts

Irina Higgins, Nicolas Sonnerat+8 more|arXiv (Cornell University)|Jul 11, 2017

Genomics and Phylogenetic StudiesBiochemistry, Genetics and Molecular Biology25 references57 citations

TL;DR

SCAN learns hierarchical, compositional visual concepts grounded in disentangled primitives using few symbol-image pairs, enabling bidirectional and recombinational multimodal inference beyond training data.

ABSTRACT

The seemingly infinite diversity of the natural world arises from a relatively small set of coherent rules, such as the laws of physics or chemistry. We conjecture that these rules give rise to regularities that can be discovered through primarily unsupervised experiences and represented as abstract concepts. If such representations are compositional and hierarchical, they can be recombined into an exponentially large set of new concepts. This paper describes SCAN (Symbol-Concept Association Network), a new framework for learning such abstractions in the visual domain. SCAN learns concepts through fast symbol association, grounding them in disentangled visual primitives that are discovered in an unsupervised manner. Unlike state of the art multimodal generative model baselines, our approach requires very few pairings between symbols and images and makes no assumptions about the form of symbol representations. Once trained, SCAN is capable of multimodal bi-directional inference, generating a diverse set of image samples from symbolic descriptions and vice versa. It also allows for traversal and manipulation of the implicit hierarchy of visual concepts through symbolic instructions and learnt logical recombination operations. Such manipulations enable SCAN to break away from its training data distribution and imagine novel visual concepts through symbolically instructed recombination of previously learnt concepts.

Motivation & Objective

• Demonstrate a framework for learning abstract, compositional visual concepts via unsupervised disentangled representations.
• Ground concepts in a small set of symbol-image pairs without strict assumptions on symbol encoding.
• Enable bidirectional inference between images and symbols (img2sym and sym2img) with diverse outputs.
• Introduce recombination operators (AND, IN COMMON, IGNORE) to navigate and extend the concept hierarchy.
• Show sample efficiency and ability to extrapolate to novel concepts beyond training data.

Proposed method

• Use beta-VAE (with a DAE-based high-level feature space) to learn disentangled visual primitives groundable by SCAN.
• Ground SCAN concepts by minimizing forward KL between the visual-primitives space and the concept space, while keeping irrelevant factors as priors.
• Represent concepts as subsets of relevant factors, enabling hierarchical abstraction and symbolic grounding.
• Train a recombination module (conditional convolution) that blends two concepts under operators AND, IN COMMON, IGNORE to generate new concept nodes.
• Allow bidirectional inference: sym2img (generate images from symbols) and img2sym (describe images with symbols).
• Evaluate with DeepMind Lab data and held-out concepts, measuring accuracy (img2sym) and diversity (variance of irrelevant factors).

Experimental results

Research questions

• RQ1 Can SCAN learn grounded, hierarchical concepts from limited symbol-image supervision?
• RQ2 Do forward KL-grounded abstractions enable diverse, accurate sym2img and img2sym inferences?
• RQ3 Can learned recombination operators navigate and extend the implicit concept hierarchy to novel concepts?
• RQ4 How does SCAN compare to baselines in accuracy and diversity, especially on unseen concepts?
• RQ5 Does SCAN maintain performance when recombining concepts at test time?

Key findings

• SCAN outperforms baselines on accuracy and diversity for sym2img generation and img2sym descriptions.
• Disentangled visual primitives ground concept space; forward KL grounding enables broad abstractions over irrelevant factors.
• Recombination operators can reach novel concepts not seen during training, preserving accuracy and diversity.
• SC A N with recombination modules maintains performance on test operators, unlike baselines.
• Partial disentanglement levels (beta) influence both accuracy and diversity, with higher disentanglement generally improving both.
• JMVAE is closest to SCAN but more prone to mode collapse without SCAN's structured latent space.