[Paper Review] Improving Bi-directional Generation between Different Modalities with Variational Autoencoders

Masahiro Suzuki, Kotaro Nakayama+1 more|arXiv (Cornell University)|Jan 26, 2018

Generative Adversarial Networks and Image Synthesis21 references3 citations

TL;DR

This paper proposes JMVAE-kl and JMVAE-h, two variants of variational autoencoders that improve bi-directional generation between high- and low-dimensional modalities—such as text and images—by preventing latent space collapse when generating high-dimensional modalities from missing inputs. The models achieve superior or comparable likelihood to unidirectional VAEs and enable controllable generation via disentangled joint representations.

ABSTRACT

We investigate deep generative models that can exchange multiple modalities bi-directionally, e.g., generating images from corresponding texts and vice versa. A major approach to achieve this objective is to train a model that integrates all the information of different modalities into a joint representation and then to generate one modality from the corresponding other modality via this joint representation. We simply applied this approach to variational autoencoders (VAEs), which we call a joint multimodal variational autoencoder (JMVAE). However, we found that when this model attempts to generate a large dimensional modality missing at the input, the joint representation collapses and this modality cannot be generated successfully. Furthermore, we confirmed that this difficulty cannot be resolved even using a known solution. Therefore, in this study, we propose two models to prevent this difficulty: JMVAE-kl and JMVAE-h. Results of our experiments demonstrate that these methods can prevent the difficulty above and that they generate modalities bi-directionally with equal or higher likelihood than conventional VAE methods, which generate in only one direction. Moreover, we confirm that these methods can obtain the joint representation appropriately, so that they can generate various variations of modality by moving over the joint representation or changing the value of another modality.

Motivation & Objective

• To address the failure of standard joint multimodal VAEs (JMVAE) in generating high-dimensional modalities (e.g., images) when their input is missing.
• To overcome the limitations of existing solutions, such as iterative sampling, which fail when the missing modality has higher dimensionality.
• To design models that maintain stable, disentangled joint representations across modalities for reliable bi-directional generation.
• To demonstrate that the proposed models achieve higher or equal likelihood than unidirectional VAEs while enabling controlled generation via latent space manipulation.

Proposed method

• JMVAE-kl introduces separate encoders for each modality and minimizes the KL divergence between their latent representations to stabilize joint representation learning.
• JMVAE-h employs a hierarchical stochastic structure in the latent space to prevent collapse by increasing representational capacity and mode separation.
• Both models extend the standard VAE framework by sharing latent variables across modalities while preserving modality-specific inference paths.
• The models use variational inference to optimize a lower bound on the joint likelihood, enabling end-to-end training via backpropagation.
• For improved image quality, JMVAE-kl is combined with GANs, using the VAE’s generative model as the GAN generator and optimizing the VAE lower bound jointly.
• Latent space manipulation is performed by interpolating or modifying latent codes, enabling controlled generation of image variations conditioned on attributes.

Experimental results

Research questions

• RQ1 Can a joint multimodal VAE (JMVAE) successfully generate high-dimensional modalities (e.g., images) when their input is missing?
• RQ2 Does the iterative sampling method effectively resolve latent collapse in high-dimensional missing modality scenarios?
• RQ3 Can JMVAE-kl and JMVAE-h prevent latent space collapse and maintain high-quality generation in both directions?
• RQ4 Do the proposed models achieve joint representations that support disentangled, controllable generation across modalities?
• RQ5 Can the joint representation be used to generate diverse image variations by manipulating latent codes or attributes?

Key findings

• JMVAE-kl and JMVAE-h significantly improve conditional log-likelihood for image generation from attributes compared to standard JMVAE, especially on the CelebA dataset.
• JMVAE-kl successfully generates attribute-conditioned facial images with high fidelity using only one sampling step, while standard JMVAE fails entirely.
• JMVAE-h improves image quality over standard JMVAE with increased sampling steps, but suffers from mode mixing, resulting in poor attribute alignment.
• The joint representation learned by JMVAE-kl enables manifold learning: images grouped by attributes form coherent clusters with average faces at the center.
• Bi-directional generation is successfully demonstrated: JMVAE-kl can generate both attributes and modified images from input images without attribute information.
• The models achieve likelihoods equal to or higher than unidirectional VAEs, confirming their effectiveness in bidirectional generation tasks.