[論文レビュー] Momentum SVGD-EM for Accelerated Maximum Marginal Likelihood Estimation

Adam Rozzio, Rafael Athanasiades+1 more|arXiv (Cornell University)|Mar 9, 2026

Generative Adversarial Networks and Image Synthesis被引用数 0

ひとこと要約

The paper introduces Momentum SVGD-EM (M-SVGD-EM), an accelerated MMLE algorithm that combines parameter-space Nesterov momentum with Wasserstein-space acceleration for SVGD-based latent variable updates, achieving faster convergence than SVGD-EM across several tasks.

ABSTRACT

Maximum marginal likelihood estimation (MMLE) can be formulated as the optimization of a free energy functional. From this viewpoint, the Expectation-Maximisation (EM) algorithm admits a natural interpretation as a coordinate descent method over the joint space of model parameters and probability measures. Recently, a significant body of work has adopted this perspective, leading to interacting particle algorithms for MMLE. In this paper, we propose an accelerated version of one such procedure, based on Stein variational gradient descent (SVGD), by introducing Nesterov acceleration in both the parameter updates and in the space of probability measures. The resulting method, termed Momentum SVGD-EM, consistently accelerates convergence in terms of required iterations across various tasks of increasing difficulty, demonstrating effectiveness in both low- and high-dimensional settings.

研究の動機と目的

• Motivate MMLE as a free-energy minimization problem and reinterpret EM as a coordinate descent over model parameters and latent-variable measures.
• Develop an accelerated, particle-based MMLE algorithm by integrating SVGD with Nesterov-inspired momentum in both parameter updates and particle updates.
• Demonstrate that the proposed M-SVGD-EM accelerates convergence across low- and high-dimensional tasks compared to existing methods.

提案手法

• Formulate MMLE as a free energy functional and use a Wasserstein gradient flow to update latent variables alongside parameter updates.
• Replace the standard SVGD-EM updates with momentum-accelerated versions: Euclidean-space Nesterov acceleration for parameters (equations 10–11) and Wasserstein-space SVGD-WNes acceleration for particles (equations 12–16, with approximation (14)).
• Compute parameter updates using θ_{t+1} = ˜θ_t + (γ/N) ∑_j ∇_θ ℓ(˜θ_t, x_t^{(j)}) followed by ˜θ_{t+1} = θ_{t+1} + α_θ(θ_{t+1} − θ_t).
• Perform accelerated particle updates via x_{t+1}^{(i)} = ˜x_t^{(i)} + (γ/N) ∑_j [k(˜x_t^{(j)}, ˜x_t^{(i)}) ∇_x ℓ(θ_{t+1}, ˜x_t^{(j)}) + ∇_1 k(˜x_t^{(j)}, ˜x_t^{(i)})], followed by ˜x_{t+1}^{(i)} = x_{t+1}^{(i)} + α_X(x_{t+1}^{(i)} − x_t^{(i)}).

(a) Parameter Estimation

実験結果

リサーチクエスチョン

• RQ1 Does applying momentum in both parameter updates and particle updates yield faster convergence in MMLE problems?
• RQ2 How does M-SVGD-EM compare to SVGD-EM and other MMLE approaches across synthetic and real datasets?
• RQ3 Can the accelerated approach maintain stability while reducing the number of iterations to convergence?

主な発見

• M-SVGD-EM consistently outperforms SVGD-EM in convergence speed across the tested tasks.
• In the Toy Hierarchical Model, higher acceleration α = 0.9 achieves the same MSE as SVGD-EM in about half the iterations and reduces average iterations to converge from 450.9±115.1 to 232±60.7.
• In Bayesian Logistic Regression on the Wisconsin dataset, M-SVGD-EM outperforms SVGD-EM and SOUL across tested accelerations, with MPGD showing competitive performance in some settings.
• In Bayesian Neural Network experiments on MNIST, M-SVGD-EM yields faster test-error reduction and tighter posterior distributions as acceleration increases.
• Across MNIST and related tasks, higher acceleration tends to lead to better predictive performance and more confident posteriors compared to SVGD-EM.

(b) MSE with $\hat{\theta}_{d_{y}}$