[Paper Review] On optimal direction gibbs sampling

J. Andrés Christen, Colin Fox+2 more|arXiv (Cornell University)|May 18, 2012

Bayesian Methods and Mixture Models1 references4 citations

TL;DR

This paper proposes an optimal direction Gibbs sampler that improves MCMC mixing by selecting sampling directions to minimize mutual information between successive states. By leveraging multivariate normal approximations and minimizing Kullback-Leibler divergence, the method achieves faster convergence and better performance than standard Gibbs or random-walk Metropolis-Hastings, especially in high-skew, non-elliptical posteriors where full conditionals are intractable.

ABSTRACT

Generalized Gibbs kernels are those that may take any direction not necessarily bounded to each axis along the parameters of the objective function. We study how to optimally choose such directions in a Directional, random scan, Gibbs sampler setting. The optimal direction is chosen by minimizing to the mutual information (Kullback-Leibler divergence) of two steps of the MCMC for a truncated Normal objective function. The result is generalized to be used when a Multivariate Normal (local) approximation is available for the objective function. Three Gibbs direction distributions are tested in highly skewed non-normal objective functions.

Motivation & Objective

• To develop a directional Gibbs sampler that improves mixing in MCMC by optimizing sampling directions.
• To address the challenge of inefficient sampling in highly skewed, non-normal posterior distributions where standard Gibbs samplers fail due to intractable full conditionals.
• To minimize mutual information between successive states as a criterion for optimal direction selection.
• To generalize the method using local multivariate normal approximations for practical implementation.
• To test performance on non-elliptical, truncated normal and strongly skewed posteriors with complex curvature.

Proposed method

• The optimal direction is selected by minimizing the mutual information between consecutive states, formulated as a Kullback-Leibler divergence between joint and product measures.
• For multivariate normal targets, the method derives closed-form expressions for the transition kernel and mutual information using the precision matrix and directional vectors.
• The algorithm uses a directional proposal distribution based on the Hessian of the log-posterior, with normalization handled via numerical integration when needed.
• An alternative direction distribution $ h_1 $ avoids normalization by using eigen-decomposition of the Hessian, but may concentrate samples in dominant directions under high skew.
• A refined distribution $ h_2 $ reweights eigenvectors by $ \lambda_i^{-b} $ with $ b \sim \text{Beta}(1,9) $, improving balance in high-contrast curvature regions.
• The method is tested on 2D examples with varying skewness and scale contrasts, comparing acceptance rates and integrated autocorrelation times (IAT).

Experimental results

Research questions

• RQ1 How can sampling directions in a Gibbs sampler be optimized to minimize dependence between successive states?
• RQ2 Can mutual information minimization lead to faster mixing in MCMC for non-elliptical, highly skewed posteriors?
• RQ3 What is the performance of the optimal direction Gibbs sampler when full conditional distributions are intractable?
• RQ4 How does the choice of direction distribution affect mixing efficiency in regions of high curvature contrast?
• RQ5 Can a reweighted eigenvector-based proposal improve robustness in skewed posteriors compared to standard directional sampling?

Key findings

• The optimal direction Gibbs sampler achieved low integrated autocorrelation times (IAT) of 2.7 to 8.8, indicating fast mixing across all test cases.
• Acceptance rates remained high (0.73–0.88), demonstrating efficient exploration without tuning difficulties.
• The $ h_2 $ distribution, which reweights eigenvectors by $ \lambda_i^{-b} $, successfully mitigated concentration in dominant directions under high skew.
• In examples with extreme curvature contrast (e.g., $ \boldsymbol{\alpha} = (-10.0, -10.0) $), $ h_2 $ reduced IAT from 8.82 to 2.71, showing significant improvement.
• The method outperformed standard Gibbs and random-walk Metropolis-Hastings in cases where full conditionals were analytically intractable.
• The approach is particularly effective in inverse problems and non-normal posteriors where conventional MCMC methods require extensive tuning.