[Paper Review] Bayesian estimators of the Gamma distribution

Alberto Llera, Christian F. Beckmann|arXiv (Cornell University)|Jul 12, 2016

Bayesian Methods and Mixture Models7 references3 citations

TL;DR

This paper proposes two novel Bayesian estimators for the Gamma distribution's shape and scale parameters, using unnormalized conjugate priors and Laplace approximations to handle the absence of a proper conjugate prior for the shape parameter. The key contribution is enabling Gamma distributions to be integrated into complex Bayesian models like variational Bayesian mixture models with controlled bias and convergence to maximum likelihood solutions.

ABSTRACT

In this paper we introduce two Bayesian estimators for learning the parameters of the Gamma distribution. The first algorithm uses a well known unnormalized conjugate prior for the Gamma shape and the second one uses a non-linear approximation to the likelihood and a prior on the shape that is conjugate to the approximated likelihood. In both cases use the Laplace approximation to compute the required expectations. We perform a theoretical comparison between maximum like- lihood and the presented Bayesian algorithms that allow us to provide non-informative parameter values for the priors hyper parameters. We also provide a numerical comparison using synthetic data. The introduction of these novel Bayesian estimators open the possibility of including Gamma distributions into more complex Bayesian structures, e.g. variational Bayesian mixture models.

Motivation & Objective

• To address the lack of a proper conjugate prior for the Gamma distribution's shape parameter, which hinders full Bayesian inference.
• To develop Bayesian estimation methods that are computationally efficient and analytically tractable for Gamma-distributed data.
• To ensure the Bayesian estimators converge to maximum likelihood solutions under non-informative hyperpriors, enabling compatibility with existing ML frameworks.
• To support integration of Gamma distributions into advanced Bayesian models, such as variational Bayesian mixture models and multivariate factorizations.
• To provide theoretical and numerical validation of bias properties and computational efficiency of the proposed estimators.

Proposed method

• Proposes a Bayesian estimator (BL1) using an unnormalized conjugate prior for the shape parameter, derived from a non-standard form involving the rate parameter and hyperparameters a, b, c.
• Uses the Laplace approximation to compute the posterior expectation of the shape parameter, avoiding intractable integrals.
• Introduces a second Bayesian estimator (BL2) that employs a non-linear approximation of the Gamma log-likelihood and a conjugate prior on the shape parameter with respect to this approximation.
• Applies the Laplace approximation to the posterior of the shape parameter in both methods, enabling closed-form expectation estimation.
• Derives posterior hyperparameters for both shape and scale parameters, with the scale parameter updated via a standard conjugate Gamma-Gamma update.
• Employs a non-informative prior setup by selecting hyperparameters such that the posterior converges to maximum likelihood estimates, particularly by setting b, c → 0+ and adjusting a accordingly.

Experimental results

Research questions

• RQ1 Can a Bayesian estimator be constructed for the Gamma distribution’s shape parameter despite the absence of a proper conjugate prior?
• RQ2 How can the Laplace approximation be effectively applied to compute posterior expectations for the shape parameter in a non-conjugate setting?
• RQ3 Do the proposed Bayesian estimators converge to maximum likelihood solutions under non-informative hyperpriors?
• RQ4 What is the computational efficiency and bias behavior of the Bayesian estimators compared to maximum likelihood methods?
• RQ5 Can these Bayesian estimators be extended to complex models such as variational Bayesian mixture models?

Key findings

• The proposed Bayesian estimators (BL1 and BL2) achieve convergence to maximum likelihood solutions when non-informative hyperpriors are used, particularly when b, c → 0+ and a is chosen appropriately.
• The computational cost of both Bayesian estimators is comparable to maximum likelihood methods, with BL2 being significantly faster than ML2 due to the use of a non-linear likelihood approximation.
• Numerical experiments show that the Bayesian estimators exhibit the same bias properties as maximum likelihood estimators, with minimal bias correction needed for small sample sizes.
• The Laplace approximation provides a stable and accurate method for estimating the posterior mean of the shape parameter, with the posterior mean closely approximating the true value in simulations.
• The posterior hyperparameters for the shape parameter (â, b̂, ĉ) are updated via closed-form expressions that depend on sufficient statistics and prior hyperparameters.
• The framework enables the integration of Gamma distributions into higher-level Bayesian models, such as variational Bayesian mixture models and multivariate factorizations, due to its analytical tractability and compatibility with conjugate updates.