[Paper Review] Bounding the Test Log-Likelihood of Generative Models

Yoshua Bengio, Li Yao+1 more|arXiv (Cornell University)|Jan 1, 2014

Algorithms and Data Compression5 citations

TL;DR

This paper proposes a more efficient, lower-bound estimator for test log-likelihood in generative models with intractable normalization constants or no analytic unnormalized density. It provides a consistent, unbiased estimator in the limit of infinite samples and a biased variant for reliable finite-sample model comparison, improving upon prior non-parametric density estimation methods.

ABSTRACT

Abstract: Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable normalization. Some of them may even not have an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model's probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood, and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing. We further propose a biased variant of the estimator that can be used reliably with a finite number of samples for the purpose of model comparison.

Motivation & Objective

• To address the challenge of evaluating generative models with intractable normalization constants or no analytic unnormalized density functions.
• To improve upon existing non-parametric density estimation methods for log-likelihood estimation in such models.
• To provide a lower bound on the true test log-likelihood that is consistent and unbiased as sample size increases.
• To develop a practical, biased variant of the estimator for reliable model comparison with finite samples.

Proposed method

• Revisits non-parametric density estimation using samples from the trained model to estimate the model's probability density.
• Proposes a more efficient estimator that maintains consistency and provides a lower bound on the true test log-likelihood.
• Proves the estimator is unbiased in the limit of infinite generated samples, even when mixing is poor.
• Introduces a biased variant that remains reliable for finite sample sizes, enabling practical model comparison.
• Uses kernel density estimation or similar non-parametric techniques to construct the density estimator from generated samples.

Experimental results

Research questions

• RQ1 Can a more efficient non-parametric estimator be constructed to bound the test log-likelihood of generative models with intractable normalization?
• RQ2 Does the proposed estimator provide a valid lower bound on the true test log-likelihood?
• RQ3 Is the estimator unbiased as the number of generated samples approaches infinity?
• RQ4 Can a biased variant of the estimator be designed for reliable model comparison with finite samples?

Key findings

• The proposed estimator provides a lower bound on the true test log-likelihood, ensuring conservative evaluation of model quality.
• The estimator is unbiased in the limit of infinite generated samples, even when the model exhibits poor mixing.
• The biased variant of the estimator is shown to be reliable for finite-sample model comparison, enabling practical use in training.
• The method improves efficiency over prior non-parametric density estimation approaches for log-likelihood evaluation.