[Paper Review] Fast Dual Variational Inference for Non-Conjugate LGMs

Mohammad Emtiyaz Khan, Aleksandr Y. Aravkin+2 more|arXiv (Cornell University)|Jun 5, 2013

Gaussian Processes and Bayesian Inference3 citations

TL;DR

This paper proposes a fast dual variational inference method for non-conjugate latent Gaussian models by reformulating the variational inference problem into a convex dual optimization with only O(N) parameters, enabling significantly faster convergence than prior methods while preserving accuracy. The approach leverages Fenchel conjugates and dual decomposition to achieve scalable inference across diverse models including Gaussian process classification and latent Gaussian Markov random fields.

ABSTRACT

Latent Gaussian models (LGMs) are widely used in statistics and machine learning. Bayesian inference in non-conjugate LGMs is difficult due to intractable integrals involving the Gaussian prior and non-conjugate likelihoods. Algorithms based on variational Gaussian (VG) approximations are widely employed since they strike a favorable balance between accuracy, generality, speed, and ease of use. However, the structure of the optimization problems associated with these approximations remains poorly understood, and standard solvers take too long to converge. We derive a novel dual variational inference approach that exploits the convexity property of the VG approximations. We obtain an algorithm that solves a convex optimization problem, reduces the number of variational parameters, and converges much faster than previous methods. Using real-world data, we demonstrate these advantages on a variety of LGMs, including Gaussian process classification, and latent Gaussian Markov random fields.

Motivation & Objective

• Address the slow convergence of standard variational inference in non-conjugate latent Gaussian models (LGMs), where exact Bayesian inference is intractable due to non-Gaussian likelihoods.
• Overcome the trade-off between computational efficiency and accuracy in existing variational Gaussian (VG) approximations, which either restrict covariance structure (losing accuracy) or suffer from slow convergence due to high-dimensional parameterization.
• Develop a generic, scalable inference method applicable beyond Gaussian processes by reducing the number of variational parameters from O(L²) to O(N) while retaining convexity.
• Enable fast and reliable inference in large-scale LGMs by exploiting convex duality and closed-form solutions via Fenchel conjugates.

Proposed method

• Formulate the variational inference problem as a dual optimization using Lagrangian relaxation of constraints linking the variational mean and covariance to the likelihood parameters.
• Introduce dual variables α and λ for the linear constraints h = Wm and ρ = diag(WVWᵀ), transforming the original non-convex problem into a convex dual via strong duality.
• Derive closed-form solutions for the primal variables (m*, V*) in terms of dual variables using the Fenchel conjugate of the negative log-likelihood terms fₙ(hₙ, ρₙ), enabling efficient optimization.
• Reduce the dual optimization to a strictly convex problem in 2N variables (α, λ), where N is the number of observations, achieving O(N) parameterization and enabling fast convergence.
• Utilize the effective domain of the Fenchel conjugate fₙ* as a natural barrier function, simplifying the optimization and ensuring numerical stability.
• Reconstruct the primal variational approximation (m*, V*) from the optimal dual variables using m* = μ − ΣWᵀα and V* = (Σ⁻¹ + Wᵀdiag(λ)W)⁻¹.

Experimental results

Research questions

• RQ1 Can we reduce the number of variational parameters in non-conjugate LGMs from O(L²) to O(N) without sacrificing convexity or convergence speed?
• RQ2 Does dual decomposition of the variational inference problem enable faster convergence compared to standard solvers and coordinate-ascent methods in non-conjugate LGMs?
• RQ3 Can the proposed method be generalized beyond Gaussian process models to other LGMs such as latent Gaussian Markov random fields and Bayesian generalized linear models?
• RQ4 How do the Fenchel conjugates of likelihood functions contribute to forming a barrier function that simplifies the dual optimization problem?

Key findings

• The proposed dual variational inference method reduces the number of optimization variables from O(L²) to O(N), where L is the latent dimension and N is the number of observations, enabling significantly faster convergence.
• The dual problem is strictly convex and involves only 2N parameters, allowing rapid convergence using standard smooth optimization methods.
• The method achieves fast convergence on real-world data, outperforming the state-of-the-art coordinate-ascent method of Khan (2012) in terms of speed, especially in non-Gaussian likelihood settings.
• The approach is generic and not restricted to Gaussian processes, successfully applied to Gaussian process classification, latent Gaussian Markov random fields, and other LGMs.
• The Fenchel conjugate fₙ* of the negative log-likelihood terms is available in closed form for common likelihoods (e.g., Bernoulli, Poisson), enabling efficient dual optimization.
• The dual formulation naturally incorporates constraints via Lagrange multipliers and ensures that the resulting variational approximation remains valid and accurate.