[Paper Review] Estimation and inference for the Wasserstein distance between mixing measures in topic models

Xin Bing, Florentina Bunea+1 more|arXiv (Cornell University)|Jun 26, 2022

Markov Chains and Monte Carlo Methods4 citations

TL;DR

This paper provides the first minimax lower bounds for estimating the Wasserstein distance between mixing measures in topic models, establishing it as a canonical metric through axiomatic justification. It further develops fully data-driven inference tools, enabling asymptotically valid confidence intervals for the Wasserstein distance in high-dimensional, potentially sparse topic models.

ABSTRACT

The Wasserstein distance between mixing measures has come to occupy a central place in the statistical analysis of mixture models. This work proposes a new canonical interpretation of this distance and provides tools to perform inference on the Wasserstein distance between mixing measures in topic models. We consider the general setting of an identifiable mixture model consisting of mixtures of distributions from a set $\mathcal{A}$ equipped with an arbitrary metric $d$, and show that the Wasserstein distance between mixing measures is uniquely characterized as the most discriminative convex extension of the metric $d$ to the set of mixtures of elements of $\mathcal{A}$. The Wasserstein distance between mixing measures has been widely used in the study of such models, but without axiomatic justification. Our results establish this metric to be a canonical choice. Specializing our results to topic models, we consider estimation and inference of this distance. Though upper bounds for its estimation have been recently established elsewhere, we prove the first minimax lower bounds for the estimation of the Wasserstein distance in topic models. We also establish fully data-driven inferential tools for the Wasserstein distance in the topic model context. Our results apply to potentially sparse mixtures of high-dimensional discrete probability distributions. These results allow us to obtain the first asymptotically valid confidence intervals for the Wasserstein distance in topic models.

Motivation & Objective

• To provide an axiomatic foundation for using the Wasserstein distance between mixing measures as a canonical metric in identifiable mixture models.
• To address the open problem of minimax lower bounds for estimating the Wasserstein distance between mixing measures in topic models when both components and weights are estimated.
• To develop fully data-driven inferential procedures, including asymptotically valid confidence intervals, for the Wasserstein distance in topic models with high-dimensional, sparse discrete distributions.
• To close the gap between existing upper bounds and optimal estimation rates by establishing the first minimax lower bounds for this estimation problem.
• To extend theoretical tools to sparse, high-dimensional settings where mixture components are discrete and potentially low-rank.

Proposed method

• Establish that the Wasserstein distance between mixing measures is the unique most discriminative convex extension of the base metric $ d $ on the set $ \mathcal{A} $, providing axiomatic justification.
• Use a reduction scheme to derive minimax lower bounds by relating the estimation of $ W(\widehat{\boldsymbol{\alpha}}, \boldsymbol{\alpha}; d) $ to $ \ell_1 $-norm estimation of $ \widehat{\alpha} - \alpha $ and $ \widehat{A} - A $.
• Leverage known results on $ \ell_1 $-estimation of sparse high-dimensional vectors and matrix components under sparsity constraints.
• Apply the reduction to two distinct subproblems: one focusing on $ \|\widehat{\alpha} - \alpha\|_1 $ with fixed components, and another on $ \|\widehat{A} - A\|_1 $ with fixed weights.
• Construct data-driven inference procedures using the derived minimax rates and asymptotic normality of estimators under regularity conditions.
• Utilize the metric $ d $ satisfying $ \min_{k \neq k'} d(A_k, A_{k'}) \geq c_d \overline{\kappa}_\tau \|A_k - A_{k'}\|_1 $ to control separation between components.

Experimental results

Research questions

• RQ1 Is the Wasserstein distance between mixing measures a canonical choice for comparing mixture models, and if so, what axiomatic justification supports this?
• RQ2 What are the minimax lower bounds for estimating the Wasserstein distance between mixing measures in topic models when both the component distributions and mixing weights are estimated?
• RQ3 Can fully data-driven confidence intervals be constructed for the Wasserstein distance between mixing measures in topic models?
• RQ4 How do the estimation rates scale with model complexity, such as sparsity $ \tau $, sample size $ n $, and dimensionality $ pK $, in high-dimensional discrete settings?
• RQ5 What is the optimal rate of convergence for estimating the Wasserstein distance between mixing measures in sparse, high-dimensional topic models?

Key findings

• The Wasserstein distance between mixing measures is uniquely characterized as the most discriminative convex extension of the base metric $ d $, providing an axiomatic foundation for its use.
• The paper establishes the first minimax lower bounds for estimating the Wasserstein distance in topic models: $ \inf_{\widehat{\boldsymbol{\alpha}}}\sup_{\alpha \in \Theta_{\alpha}(\tau), A \in \Theta_A} \mathbb{E}[W(\widehat{\boldsymbol{\alpha}}, \boldsymbol{\alpha}; d)] \gtrsim c_d \overline{\kappa}_\tau \left( \sqrt{\tau/N} + \frac{1}{\overline{\kappa}_\tau} \sqrt{pK/(nN)} \right) $.
• The lower bound confirms that near-minimax optimal estimation is achievable, complementing recent upper bounds in the literature.
• The authors construct the first fully data-driven inferential tools, enabling asymptotically valid confidence intervals for the Wasserstein distance in topic models.
• The results hold under general conditions, including potentially sparse mixtures of high-dimensional discrete distributions, with $ 1 < \tau \leq cN $ and $ pK \leq c'(nN) $.
• The theoretical framework applies to a broad class of metrics $ d $ satisfying $ \min_{k \neq k'} d(A_k, A_{k'}) \geq c_d \overline{\kappa}_\tau \|A_k - A_{k'}\|_1 $, ensuring meaningful separation between components.