[Paper Review] Convergent Expectation Propagation in Linear Models with Spike-and-slab Priors

José Miguel Hernández-Lobato, Daniel Hernández-Lobato|arXiv (Cornell University)|Dec 10, 2011

Gaussian Processes and Bayesian Inference10 references3 citations

TL;DR

This paper proposes a provably convergent Expectation Propagation (PC-EP) algorithm for linear models with spike-and-slab priors, ensuring convergence by constraining variance parameters to remain positive, which bounds the energy function minimized at each iteration. Experiments show that when standard EP fails to converge, PC-EP produces superior posterior approximations, while matching EP's performance when convergence is achieved.

ABSTRACT

Exact inference in the linear regression model with spike and slab priors is often intractable. Expectation propagation (EP) can be used for approximate inference. However, the regular sequential form of EP (R-EP) may fail to converge in this model when the size of the training set is very small. As an alternative, we propose a provably convergent EP algorithm (PC-EP). PC-EP is proved to minimize an energy function which, under some constraints, is bounded from below and whose stationary points coincide with the solution of R-EP. Experiments with synthetic data indicate that when R-EP does not converge, the approximation generated by PC-EP is often better. By contrast, when R-EP converges, both methods perform similarly.

Motivation & Objective

• To address convergence failures of standard regular sequential Expectation Propagation (R-EP) in linear models with spike-and-slab priors when training sets are very small.
• To develop a deterministic approximate inference method that guarantees convergence while preserving the accuracy of R-EP when it converges.
• To ensure the energy function minimized by the algorithm is bounded from below by constraining variance parameters to be positive.
• To provide a robust alternative to R-EP in underdetermined regression settings ($n \ll d$) where sparsity is critical for avoiding overfitting.

Proposed method

• PC-EP uses a double-loop optimization structure based on the algorithm in [3], where each outer loop minimizes an energy function related to the EP approximation.
• The method enforces constraints on the variance parameters ($\tilde{v}_{2i} > 0$) to ensure the energy function is bounded from below, a necessary condition for convergence.
• The energy function minimized by PC-EP is designed so that its stationary points coincide with the fixed points of standard R-EP, ensuring consistency with the original EP solution when convergent.
• The algorithm computes the posterior precision matrix $\mathbf{A}^{-1}$ using the Woodbury formula in $\mathcal{O}(dn^2)$ time, matching the computational cost of R-EP per iteration.
• Each iteration computes marginal variances of the posterior approximation when evaluating the gradient of the objective function, increasing the constant factor by about two orders of magnitude compared to R-EP.
• The method is designed to be compatible with fast optimization techniques (e.g., from [9]) to reduce computational overhead, though this is left for future work.

Experimental results

Research questions

• RQ1 Can a provably convergent variant of Expectation Propagation be developed for linear models with spike-and-slab priors where standard R-EP fails to converge?
• RQ2 Does constraining variance parameters to remain positive ensure convergence of the EP algorithm while preserving accuracy?
• RQ3 How does the performance of the proposed PC-EP algorithm compare to standard R-EP in terms of predictive accuracy when R-EP diverges?
• RQ4 Is the posterior approximation quality of PC-EP significantly better than R-EP in cases where R-EP fails to converge?

Key findings

• When R-EP fails to converge, the average mean square error (MSE) of PC-EP on test sets is significantly lower than R-EP’s, with PC-EP achieving an average MSE of 0.15 ± 0.04 compared to R-EP’s 0.22 ± 0.06 at $\tau = 0.9$.
• For the 17 test sets where R-EP did not converge, PC-EP achieved a lower average MSE than R-EP across all damping values $\tau \in \{0.1, 0.3, 0.5, 0.7, 0.9\}$.
• When R-EP converges (on 83–94 sets), both methods perform similarly, with PC-EP achieving an average MSE of 0.033 ± 0.007 and R-EP 0.024 ± 0.008 at $\tau = 0.9$, showing comparable predictive accuracy.
• The number of non-convergent cases increases with higher damping values, with 17 out of 100 sets failing to converge at $\tau = 0.9$, indicating that damping does not fully resolve convergence issues.
• PC-EP consistently produces better posterior approximations than R-EP in non-convergent cases, demonstrating its robustness in extreme small-sample regimes.