[Paper Review] Multimodal Controller for Generative Models

Enmao Diao, Jie Ding+1 more|arXiv (Cornell University)|Feb 7, 2020

Generative Adversarial Networks and Image Synthesis47 references4 citations

TL;DR

This paper proposes a plug-and-play multimodal controller (MC) that enables non-conditional generative models (VAE, PixelCNN, Glow, GAN) to generate high-quality, class-conditional images without adding learning parameters. The MC assigns unique, uniformly sampled subnetworks to each data modality, enabling superior generation quality and novel data modality creation through resampling, outperforming conditional baselines on CIFAR10, COIL100, and Omniglot.

ABSTRACT

Class-conditional generative models are crucial tools for data generation from user-specified class labels. Existing approaches for class-conditional generative models require nontrivial modifications of backbone generative architectures to model conditional information fed into the model. This paper introduces a plug-and-play module named `multimodal controller' to generate multimodal data without introducing additional learning parameters. In the absence of the controllers, our model reduces to non-conditional generative models. We test the efficacy of multimodal controllers on CIFAR10, COIL100, and Omniglot benchmark datasets. We demonstrate that multimodal controlled generative models (including VAE, PixelCNN, Glow, and GAN) can generate class-conditional images of significantly better quality when compared with conditional generative models. Moreover, we show that multimodal controlled models can also create novel modalities of images.

Motivation & Objective

• To develop a general-purpose, parameter-free method to convert non-conditional generative models into class-conditional ones.
• To address the limitation of existing conditional models that require additional learning parameters for conditional embeddings.
• To enable the generation of novel data modalities not seen during training using resampled subnetworks.
• To demonstrate superior performance on datasets with high intra-class variation and many modalities, such as Omniglot and COIL100.
• To provide a scalable and compatible solution across diverse backbone architectures including VAE, PixelCNN, Glow, and GAN.

Proposed method

• The multimodal controller (MC) is a nonparametric, plug-and-play module inserted at each layer of a generative model.
• Each data modality is assigned a unique, uniformly sampled subnetwork from a codebook, with no additional trainable parameters.
• The model operates by routing input through the corresponding subnetwork based on the class label, effectively creating a dedicated computation path per modality.
• Subnetworks are implemented as binary masks over the weights of the main network, enabling dynamic routing without retraining.
• Novel data modalities are created by resampling the codebook entries uniformly, generating new subnetworks not present in training.
• The method is compatible with various backbone models, including VAE, PixelCNN, Glow, and GAN, by applying the MC at each layer.

Experimental results

Research questions

• RQ1 Can a parameter-free controller be used to enable class-conditional generation in non-conditional generative models?
• RQ2 Does the multimodal controller outperform standard conditional models in terms of image quality and diversity on datasets with many modalities?
• RQ3 Can the multimodal controller generate high-fidelity, novel data modalities not present in the training data?
• RQ4 Is the performance of the multimodal controller robust across different backbone architectures (VAE, PixelCNN, Glow, GAN)?
• RQ5 How does the clustering quality of generated images on novel modalities compare between the MC and standard conditional models?

Key findings

• The multimodal controlled generative models (MCGAN, MCVAE, MCPixelCNN, MCGlow) achieved significantly better FID scores than their conditional counterparts on CIFAR10, COIL100, and Omniglot.
• On the COIL100 dataset, MCGAN successfully generated diverse object rotations, while CGAN failed to produce consistent results for all rotation modes.
• For the Omniglot dataset, MCGAN preserved both intra-class variation and inter-class distinction, whereas CGAN failed to generate certain data modalities.
• The Davies-Bouldin Index (DBI) for novel data modality creation was 2.0 (±0.30) for MCGAN on CIFAR10, compared to 33.2 (±3.46) for CGAN, indicating tighter clustering and more unbiased novel modality generation.
• The MC method achieved a DBI of 1.5 (±0.07) on COIL100 and 3.6 (±0.02) on Omniglot for novel modality creation, significantly lower than conditional baselines, confirming better clustering and fidelity.
• Qualitative results show that resampled subnetworks from the MC codebook can generate high-fidelity, diverse, and realistic novel data modalities not seen during training.