[Paper Review] Learning Generative Models using Denoising Density Estimators

Siavash Bigdeli, Geng Lin+3 more|arXiv (Cornell University)|Jan 8, 2020

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper introduces denoising density estimators (DDEs), a novel method for training generative models by directly minimizing the reverse Kullback-Leibler divergence between the true data density and the generated density. The approach uses differentiable neural networks to estimate unnormalized densities without requiring normalizing flows' invertible architectures or ODE solvers, achieving state-of-the-art log-likelihood performance on benchmark datasets while avoiding mode collapse.

ABSTRACT

Learning probabilistic models that can estimate the density of a given set of samples, and generate samples from that density, is one of the fundamental challenges in unsupervised machine learning. We introduce a new generative model based on denoising density estimators (DDEs), which are scalar functions parameterized by neural networks, that are efficiently trained to represent kernel density estimators of the data. Leveraging DDEs, our main contribution is a novel technique to obtain generative models by minimizing the KL-divergence directly. We prove that our algorithm for obtaining generative models is guaranteed to converge to the correct solution. Our approach does not require specific network architecture as in normalizing flows, nor use ordinary differential equation solvers as in continuous normalizing flows. Experimental results demonstrate substantial improvement in density estimation and competitive performance in generative model training.

Motivation & Objective

• To develop a new generative modeling framework that enables efficient density estimation and sampling without relying on invertible architectures or ODE integration.
• To address the limitations of existing methods such as mode collapse in GANs and computational complexity in normalizing flows.
• To provide a unified training objective that directly minimizes statistical divergence between generated and true data distributions.
• To demonstrate that DDEs can achieve competitive performance in both density estimation and sample generation across diverse datasets.

Proposed method

• The method uses denoising autoencoders to train scalar neural network-based density estimators (DDEs) that approximate the unnormalized density of the data distribution.
• It formulates the generator training as minimizing the reverse KL divergence between the true data density and the generated density, ensuring convergence to the correct solution.
• The DDEs are trained using a denoising objective, which allows efficient optimization without higher-order derivatives or approximations.
• The normalizing constant is estimated via importance sampling using a Gaussian proposal distribution with learned mean and variance.
• The generator is trained end-to-end using the DDEs for both real and generated data, enabling direct optimization of the energy-based model.
• The approach is architecture-agnostic, relying only on differentiability, and avoids the need for invertible flows or ODE solvers.

Experimental results

Research questions

• RQ1 Can a generative model be trained by directly minimizing the reverse KL divergence between the true and generated data densities without requiring invertible mappings or ODE solvers?
• RQ2 How does the use of denoising density estimators compare to normalizing flows and autoregressive models in terms of log-likelihood performance and sample quality?
• RQ3 Can the proposed method avoid mode collapse in high-dimensional data, such as in image generation tasks?
• RQ4 What is the impact of the noise level (ση) in the denoising objective on model performance and convergence?
• RQ5 Can the DDE-based framework be extended to high-dimensional data like CIFAR-10 or high-resolution images with minimal architectural constraints?

Key findings

• The proposed method achieves state-of-the-art log-likelihood results on four benchmark datasets: POWER (0.97 ± 0.18), GAS (9.73 ± 1.14), HEPMASS (-11.3 ± 0.16), and MINIBOON (-6.94 ± 1.81).
• The model demonstrates robustness to mode collapse, performing comparably to state-of-the-art Wasserstein GANs on stacked MNIST.
• The method achieves competitive performance without requiring normalizing flows’ invertible coupling layers or ODE integration, simplifying training and inference.
• The approach shows strong generalization across diverse data types, including high-dimensional tabular and image data.
• Importance sampling with 51,200 samples per run yields stable estimates of the normalizing constant, with low variance across experiments.
• Theoretical analysis confirms that the training algorithm converges to a unique global optimum under the reverse KL divergence objective.