[Paper Review] Borrowing strength in hierarchical Bayes: convergence of the Dirichlet base measure

XuanLong Nguyen|arXiv (Cornell University)|Jan 4, 2013

Statistical Methods and Inference3 citations

TL;DR

This paper establishes convergence rates for the base measure in hierarchical Dirichlet processes using transportation distances under geometric sparsity conditions. It demonstrates that borrowing strength across multiple groups enables a dramatic improvement in estimation efficiency, shifting convergence from nonparametric to parametric rates.

ABSTRACT

This paper studies posterior concentration behavior of the base probability measure $G$ of a Dirichlet measure $\mathcal{D}_{\alpha G}$, given observations associated with $m$ Dirichlet processes sampled from $\mathcal{D}_{\alpha G}$, as $m$ and the number of observations $m imes n$ tend to infinity. The base measure itself is endowed with another Dirichlet prior, a construction known as the hierarchical Dirichlet processes (Teh et al, 2006). Convergence rates are established in transportation distances (i.e. Wasserstein metrics) under various geometrically sparse conditions on the support of the true base measure. As a consequence of the theory we demonstrate the benefit of borrowing strength in the inference of multiple groups of data --- a heuristic argument commonly used to motivate hierarchical modeling. In certain settings, the gain in efficiency due to the latent hierarchy can be dramatic, improving from a standard nonparametric rate to a parametric rate of convergence. Tools developed include transportation distances for nonparametric Bayesian hierarchies of random measures, the existence of tests for Dirichlet measures, and geometric properties of the support of Dirichlet measures.

Motivation & Objective

• To analyze posterior concentration behavior of the base measure in a hierarchical Dirichlet process as data size increases.
• To formalize the heuristic of 'borrowing strength' across groups in nonparametric Bayesian modeling using rigorous convergence theory.
• To establish convergence rates in Wasserstein metrics for the base measure under geometrically sparse support conditions.
• To develop tools for testing Dirichlet measures and analyzing geometric properties of their supports in hierarchical nonparametric models.

Proposed method

• Uses transportation distances (Wasserstein metrics) to measure convergence of the base measure in the hierarchical Dirichlet process framework.
• Endows the base measure $ G $ with a Dirichlet prior, forming a hierarchical Dirichlet process as in Teh et al. (2006).
• Analyzes asymptotic behavior as both the number of groups $ m $ and total observations $ m imes n $ tend to infinity.
• Applies geometric sparsity conditions on the true base measure's support to derive convergence rates.
• Employs tools such as existence of tests for Dirichlet measures and geometric analysis of support structures.
• Establishes theoretical convergence rates under regularity and sparsity assumptions on the true base measure.

Experimental results

Research questions

• RQ1 How does the posterior concentration of the base measure in a hierarchical Dirichlet process behave as data from multiple groups accumulate?
• RQ2 What conditions on the support of the true base measure ensure fast convergence rates in transportation distance?
• RQ3 To what extent does borrowing strength across groups improve estimation efficiency in nonparametric Bayesian models?
• RQ4 Can the hierarchical structure lead to parametric convergence rates instead of the standard nonparametric rates?
• RQ5 What theoretical tools are necessary to analyze convergence in nonparametric Bayesian hierarchies of random measures?

Key findings

• Under geometric sparsity conditions on the true base measure's support, the posterior distribution of the base measure converges at a rate that can be parametric rather than nonparametric.
• The hierarchical structure enables significant efficiency gains by borrowing strength across multiple groups, leading to faster convergence than independent nonparametric inference.
• Convergence rates are established in Wasserstein metrics, providing a rigorous foundation for comparing distributions in the hierarchical model.
• The existence of tests for Dirichlet measures is proven, supporting the theoretical analysis of posterior concentration.
• Geometric properties of the support of Dirichlet measures are shown to play a critical role in determining convergence behavior.
• The theory formally justifies the common heuristic of 'borrowing strength' in hierarchical modeling, showing it can yield parametric rates of convergence in favorable settings.