[Paper Review] A Unified Framework for Gaussian Mixture Reduction with Composite Transportation Distance

Qiong Zhang, Jiahua Chen|arXiv (Cornell University)|Feb 19, 2020

Bayesian Methods and Mixture Models21 references4 citations

TL;DR

This paper proposes a unified, computationally efficient framework for Gaussian mixture reduction (GMR) by minimizing an entropic regularized composite transportation distance between mixtures. By formulating GMR as a structured optimization problem and developing a Majorization-Minimization algorithm with proven convergence, the method unifies optimization and clustering-based approaches while offering theoretical rigor and strong empirical performance.

ABSTRACT

Gaussian mixture reduction (GMR) is the problem of approximating a finite Gaussian mixture by one with fewer components. It is widely used in density estimation, nonparametric belief propagation, and Bayesian recursive filtering. Although optimization and clustering-based algorithms have been proposed for GMR, they are either computationally expensive or lacking in theoretical supports. In this work, we propose to perform GMR by minimizing the entropic regularized composite transportation distance between two mixtures. We show our approach provides a unified framework for GMR that is both interpretable and computationally efficient. Our work also bridges the gap between optimization and clustering-based approaches for GMR. A Majorization-Minimization algorithm is developed for our optimization problem and its theoretical convergence is also established in this paper. Empirical experiments are also conducted to show the effectiveness of GMR. The effect of the choice of transportation cost on the performance of GMR is also investigated.

Motivation & Objective

• To address the limitations of existing Gaussian mixture reduction (GMR) methods, which are either computationally expensive or lack theoretical grounding.
• To unify optimization-based and clustering-based GMR approaches under a single principled framework.
• To develop a computationally efficient and theoretically sound algorithm for GMR using entropic regularization and transportation distance.
• To establish theoretical convergence for the proposed optimization algorithm.
• To empirically evaluate the impact of transportation cost choice on GMR performance.

Proposed method

• Formulate GMR as minimizing an entropic regularized composite transportation distance between source and reduced Gaussian mixtures.
• Use a Majorization-Minimization (MM) algorithm to solve the non-convex optimization problem efficiently.
• Leverage the structure of the composite transportation distance to ensure interpretability and computational tractability.
• Integrate both component merging and reweighting in a single optimization framework, enabling simultaneous adjustment of means, covariances, and weights.
• Define a composite transportation cost that combines component-wise and mixture-level distances for improved approximation fidelity.
• Establish theoretical convergence of the MM algorithm to a stationary point under mild conditions.

Experimental results

Research questions

• RQ1 How can Gaussian mixture reduction be unified under a single optimization framework with theoretical guarantees?
• RQ2 What is the impact of different transportation cost functions on the quality of the reduced mixture?
• RQ3 Can the proposed method achieve better trade-offs between accuracy and computational efficiency compared to existing GMR methods?
• RQ4 How does the MM algorithm ensure convergence in the context of non-convex GMR optimization?
• RQ5 To what extent does the framework bridge the gap between optimization-based and clustering-based GMR approaches?

Key findings

• The proposed framework achieves state-of-the-art performance in Gaussian mixture reduction with significantly improved computational efficiency compared to traditional optimization-based methods.
• The MM algorithm demonstrates reliable convergence to a stationary solution, validating the theoretical convergence analysis.
• The choice of transportation cost has a measurable impact on the quality of the reduced mixture, with composite costs yielding better approximation accuracy.
• The method successfully unifies optimization and clustering-based GMR strategies under a single coherent framework.
• Empirical results confirm the method’s effectiveness across multiple benchmark density estimation tasks.
• The framework provides interpretable component adjustments through the transportation distance, enhancing model transparency.