[Paper Review] Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models

Yuge Shi, N. Siddharth+2 more|arXiv (Cornell University)|Nov 8, 2019

Generative Adversarial Networks and Image Synthesis55 references91 citations

TL;DR

Proposes MMVAE, a multimodal variational autoencoder with a mixture-of-experts posterior to jointly model and generate across modalities, enabling latent factorisation, coherent joint and cross-generation, and improved modality-specific learning. It outperforms PoE-based MVAE on image–image and image–language tasks.

ABSTRACT

Learning generative models that span multiple data modalities, such as vision and language, is often motivated by the desire to learn more useful, generalisable representations that faithfully capture common underlying factors between the modalities. In this work, we characterise successful learning of such models as the fulfillment of four criteria: i) implicit latent decomposition into shared and private subspaces, ii) coherent joint generation over all modalities, iii) coherent cross-generation across individual modalities, and iv) improved model learning for individual modalities through multi-modal integration. Here, we propose a mixture-of-experts multimodal variational autoencoder (MMVAE) to learn generative models on different sets of modalities, including a challenging image-language dataset, and demonstrate its ability to satisfy all four criteria, both qualitatively and quantitatively.

Motivation & Objective

• Define four criteria for successful multi-modal generative models: latent factorisation into shared/private subspaces, coherent joint generation, coherent cross-generation, and synergy improving single-modality performance.
• Introduce MMVAE with a mixture-of-experts variational posterior over modalities to learn multi-modal representations.
• Demonstrate MMVAE on image–image (MNIST–SVHN) and image–language (CUB captions) tasks, including challenging image↔language transformation.
• Compare MMVAE to PoE-based MVAE and analyze latent structure and generation coherence.

Proposed method

• Use a variational autoencoder framework with a joint generative model p(z, x1:M) = p(z) ∏m p(xm|z).
• Approximate the joint posterior q(z|x1:M) with a mixture of unimodal posteriors: q(z|x1:M) = Σm αm qφm(z|xm), where αm = 1/M.
• Adopt an IWAE-style bound extended to multiple modalities to obtain a tighter objective: LMoE-IWAE with stratified sampling over modalities.
• Compare MoE vs Product-of-Experts (PoE) factorisations and argue MoE yields better latent factorisation and cross-generation in settings with all modalities present during training.
• Provide training with Laplace priors/posteriors to encourage axis-aligned latent representations; use Adam/AMSGrad for optimization.
• Evaluate using both qualitative generations and quantitative metrics including cross-generation/coherence and latent space linear separability.

Experimental results

Research questions

• RQ1 Can a mixture-of-experts variational posterior enable latent factorisation that separates shared and private modality information?
• RQ2 Do MMVAE models produce coherent joint generations across modalities and coherent cross-generations between modalities?
• RQ3 Does multi-modal training improve single-modality generation rather than hinder it (synergy)?
• RQ4 How does MMVAE compare to PoE-based MVAE in terms of cross-generation coherence and latent representation quality?
• RQ5 Are image↔language transformations feasible and coherent within a single MMVAE framework?

Key findings

• MMVAE achieves better latent factorisation than single-modality VAEs and MVAE with PoE, as evidenced by more discriminative latent representations for digits across MNIST/SVHN.
• Joint generation coherence shows higher cross-modal alignment with MMVAE than MVAE, demonstrated by higher cross-generation coherence metrics.
• Cross-generation results indicate MMVAE can generate semantically consistent data across modalities (e.g., MNIST digits conditioned on SVHN, and vice versa).
• On CUB, MMVAE yields joint image-caption coherence, with cross-generation producing captions aligned to image content and vice versa.
• Compared to MVAE, MMVAE yields higher joint-generation correlations in CUB and MNIST–SVHN tasks, indicating improved multi-modal integration and utilization of information across modalities.