[Paper Review] Conditional Generative Moment-Matching Networks

Yong Ren, Jun Zhu+2 more|arXiv (Cornell University)|Jan 1, 2016

Topic Modeling12 citations

TL;DR

This paper proposes Conditional Generative Moment-Matching Networks (CGMMN), a deep generative model that learns conditional distributions using a conditional maximum mean discrepancy (CMMD) criterion, optimized via stochastic gradient descent with back-propagation. CGMMN achieves competitive performance in predictive modeling, contextual generation, and Bayesian distillation by effectively matching conditional moment distributions.

ABSTRACT

Maximum mean discrepancy (MMD) has been successfully applied to learn deep generative models for characterizing a joint distribution of variables via kernel mean embedding. In this paper, we present conditional generative moment-matching networks (CGMMN), which learn a conditional distribution given some input variables based on a conditional maximum mean discrepancy (CMMD) criterion. The learning is performed by stochastic gradient descent with the gradient calculated by back-propagation. We evaluate CGMMN on a wide range of tasks, including predictive modeling, contextual generation, and Bayesian dark knowledge, which distills knowledge from a Bayesian model by learning a relatively small CGMMN student network. Our results demonstrate competitive performance in all the tasks.

Motivation & Objective

• To develop a deep generative model that learns conditional distributions given input variables.
• To extend maximum mean discrepancy (MMD) to the conditional setting via conditional MMD (CMMD) for improved distribution matching.
• To enable end-to-end training of conditional generative models using stochastic gradient descent with back-propagation.
• To evaluate the model on diverse tasks, including predictive modeling, contextual generation, and knowledge distillation from Bayesian models.
• To demonstrate that CGMMN can effectively distill knowledge from complex Bayesian models into compact student networks.

Proposed method

• The method formulates conditional distribution learning as minimizing a conditional maximum mean discrepancy (CMMD) between the true and generated conditional distributions.
• It employs kernel mean embeddings to represent conditional distributions in a reproducing kernel Hilbert space (RKHS).
• The CMMD loss is optimized using stochastic gradient descent with gradients computed via back-propagation through the generative network.
• The model uses a deep neural network to parameterize the conditional generator, mapping input variables to output samples.
• The training objective ensures that the conditional moment distributions of generated data match those of real data across multiple kernel functions.
• The framework supports both unconditional and conditional generation by conditioning the generator on input variables.

Experimental results

Research questions

• RQ1 Can CMMD be effectively extended to model conditional distributions in deep generative models?
• RQ2 How well does CGMMN perform in learning conditional distributions compared to existing methods?
• RQ3 Can CGMMN be used to distill knowledge from large Bayesian models into smaller student networks?
• RQ4 Does CGMMN achieve competitive performance in predictive modeling and contextual generation tasks?
• RQ5 How robust is CGMMN to distribution shifts and complex conditional dependencies?

Key findings

• CGMMN achieves competitive performance in predictive modeling tasks by accurately learning conditional distributions from data.
• The model demonstrates strong capabilities in contextual generation, producing diverse and realistic samples conditioned on input variables.
• CGMMN successfully performs Bayesian dark knowledge distillation, transferring knowledge from a Bayesian model to a compact student network.
• The use of CMMD enables effective matching of conditional moment distributions, leading to stable and efficient training.
• Empirical results show that CGMMN generalizes well across diverse tasks, including those with complex conditional structures.
• The method achieves state-of-the-art or competitive results without requiring explicit likelihood computation or complex inference schemes.