[Paper Review] MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learning

Miguel Vasco, Francisco S. Melo+1 more|arXiv (Cornell University)|Jun 4, 2020

Topic Modeling29 references4 citations

TL;DR

MHVAE proposes a human-inspired hierarchical variational autoencoder that learns modality-specific and joint multimodal representations for robust cross-modality inference. By using modality-specific representation dropout to approximate the joint posterior, it achieves state-of-the-art performance in modality-specific reconstruction and cross-modality generation with minimal computational cost.

ABSTRACT

Humans are able to create rich representations of their external reality. Their internal representations allow for cross-modality inference, where available perceptions can induce the perceptual experience of missing input modalities. In this paper, we contribute the Multimodal Hierarchical Variational Auto-encoder (MHVAE), a hierarchical multimodal generative model for representation learning. Inspired by human cognitive models, the MHVAE is able to learn modality-specific distributions, of an arbitrary number of modalities, and a joint-modality distribution, responsible for cross-modality inference. We formally derive the model's evidence lower bound and propose a novel methodology to approximate the joint-modality posterior based on modality-specific representation dropout. We evaluate the MHVAE on standard multimodal datasets. Our model performs on par with other state-of-the-art generative models regarding joint-modality reconstruction from arbitrary input modalities and cross-modality inference.

Motivation & Objective

• To develop a multimodal generative model that mimics human cognitive processing by learning hierarchical representations across multiple sensory modalities.
• To enable cross-modality inference—reconstructing missing modalities from partial inputs—by jointly modeling modality-specific and joint distributions.
• To address the limitations of single-latent-space models that compromise representation capacity by forcing all modalities into one shared space.
• To propose an efficient posterior approximation method for the joint-modality distribution that scales to arbitrary numbers of modalities.
• To evaluate the model on standard benchmarks and demonstrate superior performance in reconstruction and cross-modality generation.

Proposed method

• The MHVAE employs a hierarchical architecture with modality-specific encoders and a shared joint-encoding layer, inspired by the Convergence-Divergence Zone (CDZ) cognitive model.
• It uses a variational inference framework to optimize the evidence lower bound (ELBO), formally derived for the hierarchical multimodal setup.
• A novel posterior approximation method is introduced: modality-specific representation dropout, which enables effective joint posterior estimation without additional inference networks.
• The model trains end-to-end using backpropagation, with the dropout mechanism promoting cross-modality generalization during optimization.
• The architecture supports arbitrary numbers of input modalities and maintains modality-specific representation capacity while enabling joint inference.
• The method avoids expensive pairwise divergence terms by leveraging hierarchical structure and dropout-based posterior approximation.

Experimental results

Research questions

• RQ1 Can a hierarchical generative model outperform single-latent-space models in multimodal representation learning?
• RQ2 How can cross-modality inference be effectively learned and optimized in a scalable way across multiple modalities?
• RQ3 Can modality-specific representation dropout serve as an effective and efficient method for approximating the joint-modality posterior?
• RQ4 Does the MHVAE achieve better reconstruction and cross-modality generation than state-of-the-art models on standard benchmarks?
• RQ5 How does the dimensionality of representation spaces affect performance on complex datasets like CelebA?

Key findings

• The MHVAE achieves state-of-the-art performance in modality-specific reconstruction on MNIST and FashionMNIST, even with lower-dimensional representation spaces.
• On the CelebA dataset, the MHVAE performs on par with other state-of-the-art models in joint-modality reconstruction and cross-modality inference.
• The MHVAE outperforms the MVAE model in cross-modality inference performance, as measured by label-based evaluation.
• The proposed modality-specific representation dropout method enables efficient joint posterior approximation with minimal computational overhead.
• The hierarchical design preserves modality-specific representation capacity while enabling robust cross-modality generation under incomplete input conditions.
• The model demonstrates strong generalization and scalability to arbitrary numbers of modalities due to its modular architecture and efficient posterior approximation.