[Paper Review] Generative Models of Visually Grounded Imagination

Ramakrishna Vedantam, Ian Fischer+2 more|arXiv (Cornell University)|May 30, 2017

Multimodal Machine Learning ApplicationsComputer Science21 references50 citations

TL;DR

The paper extends variational autoencoders to jointly model images and attribute descriptions, enabling generation from partially specified concepts via a novel TELBO objective and a product-of-experts inference network, with evaluation using the 3 C’s (correctness, coverage, compositionality) on MNIST-A and CelebA.

ABSTRACT

It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before. We call the ability to create images of novel semantic concepts visually grounded imagination. In this paper, we show how we can modify variational auto-encoders to perform this task. Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially specified (abstract) concepts in a principled and efficient way. We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C's). Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of Suzuki et.al. and the BiVCCA method of Wang et.al.) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset.

Motivation & Objective

• Enable generation of images from abstract or partially specified attribute concepts using a joint image-attribute VAE framework.
• Introduce a novel training objective (TELBO) for paired data and flexible inference networks for fully and partially observed inputs.
• Handle missing attributes at test time via a product-of-experts posterior to maintain well-conditioned latent representations.
• Propose objective evaluation metrics (the 3 C’s) to quantify correctness, coverage, and compositionality of generated images.
• Compare against existing joint VAE methods on MNIST-with-attributes and CelebA datasets to demonstrate improvements.

Proposed method

• Define a joint generative model p(x, y, z) = p(z) p(x|z) p(y|z) with y represented as a vector of attributes.
• Extend VAE training to TELBO, optimizing a triple ELBO (TELBO) to jointly train image and attribute decoders with a shared latent space.
• Use three inference networks q(z|x,y), q(z|x), q(z|y) to enable test-time inference with paired and unpaired data.
• Implement a product-of-experts (POE) posterior q(z|y_O) ∝ p(z) ∏_{k∈O} q(z|y_k) to handle partially observed attribute sets.
• Train unimodal posteriors and decoders while freezing the decoders, enabling joint optimization of the TELBO terms.
• Introduce a compositional abstraction hierarchy over attributes to generate images at varying levels of granularity.
• Propose evaluation metrics (3 C’s) based on a fixed attribute classifier to assess correctness, coverage, and compositionality.

Experimental results

Research questions

• RQ1 How can we extend VAEs to jointly model images and attribute vectors in a multimodal setting?
• RQ2 Can a product-of-experts posterior effectively handle partially specified (abstract) attribute concepts during inference?
• RQ3 Do the proposed TELBO objectives enable robust learning and generation across different levels of abstraction and missing data?
• RQ4 How can we quantify the quality of visually grounded imagination in terms of correctness, coverage, and compositionality?
• RQ5 Do the proposed methods outperform existing joint VAE approaches on benchmark datasets like MNIST-A and CelebA?

Key findings

• TELBO-based JVAE with POE inference achieves competitive or superior correctness and coverage compared to BiVCCA and JMVAE on MNIST-A and CelebA.
• The POE posterior makes the latent space conditioning adaptive: more attributes lead to a narrower posterior, enabling diverse yet accurate generations.
• The 3 C’s (correctness, coverage, compositionality) provide a practical, objective evaluation framework for conditional image generation from abstract concepts.
• Experiments on MNIST-A confirm that TELBO and JMVAE produce high-quality, attribute-consistent images, with BiVCCA producing blurrier outputs.
• The approach supports missing data at test time, maintaining well-conditioned posteriors and plausible generations across varying attribute completeness.
• Compared with related joint-VAE methods, the proposed model better handles abstraction levels and compositional queries, demonstrating richer generative capabilities.