[Paper Review] Multimodal Generative Models for Scalable Weakly-Supervised Learning

Mike Wu, Noah D. Goodman|arXiv (Cornell University)|Feb 14, 2018

Multimodal Machine Learning Applications22 references150 citations

TL;DR

The paper introduces MVAE, a multimodal variational autoencoder using a product-of-experts inference network and sub-sampled training to learn a joint distribution across arbitrary modalities with missing data, achieving state-of-the-art performance with fewer parameters and enabling weakly-supervised learning.

ABSTRACT

Multiple modalities often co-occur when describing natural phenomena. Learning a joint representation of these modalities should yield deeper and more useful representations. Previous generative approaches to multi-modal input either do not learn a joint distribution or require additional computation to handle missing data. Here, we introduce a multimodal variational autoencoder (MVAE) that uses a product-of-experts inference network and a sub-sampled training paradigm to solve the multi-modal inference problem. Notably, our model shares parameters to efficiently learn under any combination of missing modalities. We apply the MVAE on four datasets and match state-of-the-art performance using many fewer parameters. In addition, we show that the MVAE is directly applicable to weakly-supervised learning, and is robust to incomplete supervision. We then consider two case studies, one of learning image transformations---edge detection, colorization, segmentation---as a set of modalities, followed by one of machine translation between two languages. We find appealing results across this range of tasks.

Motivation & Objective

• Motivate learning a joint multimodal representation when data comes in multiple co-occurring modalities.
• Develop an efficient inference scheme that scales to many modalities with missing data.
• Enable learning under weak supervision by leveraging uni-modal data to inform a joint distribution.
• Demonstrate the approach on diverse datasets and case studies including vision-language and transformations.

Proposed method

• Formulate a multimodal VAE with a joint latent variable z and conditionally independent modalities given z.
• Derive a product-of-experts (PoE) posterior q(z|X) using uni-modal posteriors, with a prior expert, to handle any subset of observed modalities.
• Provide a Gaussian closed-form for PoE when both prior and uni-modal posteriors are Gaussian.
• Introduce a sub-sampled training objective that optimizes ELBO terms for full, single-modality, and a random subset of multi-modal subsets to manage missing data and training efficiency.
• Share parameters across modality combinations so the same uni-modal encoders can be reused regardless of which modalities are present.
• Demonstrate robustness to incomplete supervision by training on partially paired data and using it for weakly-supervised learning tasks.

Experimental results

Research questions

• RQ1 Can MVAE learn a coherent joint distribution over multiple modalities with missing data using a single, shared inference structure?
• RQ2 Does product-of-experts inference improve stability and performance over alternative multi-modal inference schemes?
• RQ3 How does the MVAE perform under weak supervision when only a subset of examples are fully multi-modal?
• RQ4 Can the MVAE scale to many modalities and still learn useful joint representations across diverse tasks?

Key findings

• MVAE matches state-of-the-art performance on MNIST, binarized MNIST, MultiMNIST, FashionMNIST, and CelebA using fewer parameters.
• MVAE with 18+ modalities (CelebA attributes as separate modalities) benefits from shared statistical strength.
• MVAE effectively learns joint multimodal transformations by treating transformations like colorization, edge detection, and segmentation as modalities.
• Under weak supervision, MVAE outperforms baselines and remains competitive with fully supervised models in middle-data regimes.
• The MVAE is robust to missing modalities and supports learning translations between languages with limited paired data (weakly supervised translation).
• Variance analyses show MVAE achieves lower importance-weight variance than prior joint-modal models, indicating stronger inference networks.