[Paper Review] Deep Generative Models with Learnable Knowledge Constraints

Zhiting Hu, Zichao Yang+5 more|arXiv (Cornell University)|Jun 26, 2018

Topic Modeling40 citations

TL;DR

The paper reframes posterior regularization (PR) as a form of reinforcement learning to learn constraints that guide deep generative models, enabling model-agnostic, learnable knowledge constraints for both image and text generation. It proposes an RL-based algorithm to learn constraint functions f_phi and to train generative models p_theta, including implicit models.

ABSTRACT

The broad set of deep generative models (DGMs) has achieved remarkable advances. However, it is often difficult to incorporate rich structured domain knowledge with the end-to-end DGMs. Posterior regularization (PR) offers a principled framework to impose structured constraints on probabilistic models, but has limited applicability to the diverse DGMs that can lack a Bayesian formulation or even explicit density evaluation. PR also requires constraints to be fully specified a priori, which is impractical or suboptimal for complex knowledge with learnable uncertain parts. In this paper, we establish mathematical correspondence between PR and reinforcement learning (RL), and, based on the connection, expand PR to learn constraints as the extrinsic reward in RL. The resulting algorithm is model-agnostic to apply to any DGMs, and is flexible to adapt arbitrary constraints with the model jointly. Experiments on human image generation and templated sentence generation show models with learned knowledge constraints by our algorithm greatly improve over base generative models.

Motivation & Objective

• Motivate the integration of rich, structured domain knowledge into diverse deep generative models (DGMs) beyond fully specified priors.
• Extend posterior regularization (PR) to support learnable constraints that adapt during training.
• Leverage reinforcement learning insights, notably maximum entropy IRL, to learn constraint functions from data.
• Develop a practical, model-agnostic algorithm that works with implicit and explicit DGMs.
• Demonstrate improved generation in image (pose-conditioned person image) and text (template-guided) tasks.

Proposed method

• Establish mathematical correspondence between PR and entropy-regularized RL, and between PR and MaxEnt IRL.
• Represent constraints as learnable functions f_phi(x) and optimize via an EM-style procedure.
• Treat learning of f_phi as reward learning in MaxEnt IRL, using gradient estimates that leverage importance sampling with p_theta as proposal.
• When p_theta is implicit, adopt reverse KL minimization to avoid density evaluation, aligning with adversarial training intuition.
• Iteratively update constraint parameters phi (via Eq. 8) and generative model parameters theta (via Eq. 12 or Eq. 10, depending on density evaluability).
• Relate the framework to energy-based models and GANs, highlighting how q_phi approximates an energy-based distribution and how the generator seeks to fool the constraint.

Experimental results

Research questions

• RQ1 Can learnable, structured constraints be jointly learned with deep generative models to encode domain knowledge without requiring fully specified priors?
• RQ2 Does a PR–RL correspondence enable practical, scalable learning of such constraints for both implicit and explicit DGMs?
• RQ3 Do learned, structure-preserving constraints improve generation quality in image and text tasks compared to base models and fixed constraints?
• RQ4 How can MaxEnt IRL techniques be used to learn constraint functions from demonstrations or data distributions in the context of DGMs?

Key findings

• Learning constraints improves generation quality over base models in both image and text tasks.
• In pose-conditioned person image generation, a learned body-part consistency constraint yields higher SSIM and human preference than base and fixed-constraint baselines (SSIM: 0.727 with learned constraint vs 0.676 base; human: 0.77 vs 0.03).
• In template-guided sentence generation, learned constraints improve perplexity and human judgments over base models and direct constraint learning.
• The approach remains effective for implicit models by using reverse KL objectives and importance-sampling-based gradient estimates.
• The learned constraint acts analogously to a discriminator in adversarial settings, but the framework maintains a model-agnostic stance focused on improving the generator.
• Experimental results demonstrate smooth convergence and stable training with learned constraints.