[Paper Review] Variational Autoencoder for Deep Learning of Images, Labels and Captions

Yunchen Pu, Zhe Gan+5 more|arXiv (Cornell University)|Sep 28, 2016

Generative Adversarial Networks and Image Synthesis35 references372 citations

TL;DR

The paper proposes a variational autoencoder framework to jointly model images, their labels, and captions, enabling deep learning across multiple modalities.

ABSTRACT

A novel variational autoencoder is developed to model images, as well as associated labels or captions. The Deep Generative Deconvolutional Network (DGDN) is used as a decoder of the latent image features, and a deep Convolutional Neural Network (CNN) is used as an image encoder; the CNN is used to approximate a distribution for the latent DGDN features/code. The latent code is also linked to generative models for labels (Bayesian support vector machine) or captions (recurrent neural network). When predicting a label/caption for a new image at test, averaging is performed across the distribution of latent codes; this is computationally efficient as a consequence of the learned CNN-based encoder. Since the framework is capable of modeling the image in the presence/absence of associated labels/captions, a new semi-supervised setting is manifested for CNN learning with images; the framework even allows unsupervised CNN learning, based on images alone.

Motivation & Objective

• Motivate the need for unified modeling of images, labels, and captions in a single probabilistic framework.
• Develop a variational autoencoder architecture that can handle multimodal outputs (images and text)
• Enable joint learning of visual and textual representations to improve generative and discriminative capabilities.
• Provide a learning objective and optimization approach that integrates image and caption data within VAE.
• Demonstrate the feasibility and potential benefits of multimodal VAE for combined vision-language tasks.

Proposed method

• Introduce a variational autoencoder setup for jointly modeling images and captions (and optional labels).
• Define encoder and decoder networks that map between image space and latent representations, and between latent representations and caption sequences.
• Leverage a variational lower bound (ELBO) as the training objective to jointly optimize image reconstruction and caption generation.
• Incorporate mechanisms to align latent representations across modalities to enable coherent multimodal generation.
• Discuss training details and architectural choices that enable end-to-end learning of image-label-caption triplets.

Experimental results

Research questions

• RQ1 Can a single variational framework jointly model images, labels, and captions effectively?
• RQ2 How does joint multimodal training influence the quality of generated images and captions compared to modality-specific VAEs?
• RQ3 What is the impact of incorporating labels into the VAE on inference and generation of captions?
• RQ4 What architectural or objective adjustments are beneficial for aligning multimodal latent spaces?

Key findings

• The proposed multimodal VAE framework demonstrates feasibility for jointly learning images, labels, and captions.
• Experimental validation suggests it is possible to generate coherent images and captions from a shared latent space.
• The approach provides a unified probabilistic model that can capture relationships between visual content and textual descriptions.
• The work discusses architectural choices and training strategies that support multimodal integration in VAEs.