[Paper Review] Efficient hierarchical clustering for continuous data

Ricardo Henao, Joseph E. Lucas|arXiv (Cornell University)|Apr 20, 2012

Gaussian Processes and Bayesian Inference17 references3 citations

TL;DR

This paper proposes an efficient sequential Monte Carlo (SMC) sampler for Bayesian hierarchical clustering of continuous, non-i.i.d. data using coalescent priors and Gaussian process modeling. It achieves quadratic computational complexity, significantly reducing runtime over the original cubic-time sampler while maintaining high performance, and introduces a refined greedy algorithm that outperforms prior methods, especially on small datasets.

ABSTRACT

We present an new sequential Monte Carlo sampler for coalescent based Bayesian hierarchical clustering. Our model is appropriate for modeling non-i.i.d. data and offers a substantial reduction of computational cost when compared to the original sampler without resorting to approximations. We also propose a quadratic complexity approximation that in practice shows almost no loss in performance compared to its counterpart. We show that as a byproduct of our formulation, we obtain a greedy algorithm that exhibits performance improvement over other greedy algorithms, particularly in small data sets. In order to exploit the correlation structure of the data, we describe how to incorporate Gaussian process priors in the model as a flexible way to model non-i.i.d. data. Results on artificial and real data show significant improvements over closely related approaches.

Motivation & Objective

• To address the lack of principled Bayesian hierarchical clustering methods for non-i.i.d. continuous data.
• To reduce the computational cost of Bayesian hierarchical clustering from cubic to quadratic complexity without approximation.
• To improve upon the greedy algorithm of Teh et al. (2008) by deriving a corrected version that performs better on small datasets.
• To incorporate Gaussian process priors to model correlation structures in continuous data.
• To develop a scalable inference method that enables use of hierarchical clustering as a building block in larger models.

Proposed method

• Proposes a sequential Monte Carlo (SMC) sampler that efficiently samples from the posterior distribution over hierarchical tree structures.
• Uses Kingman’s coalescent as a non-informative prior over binary tree topologies and merging times.
• Incorporates Gaussian process priors to model non-i.i.d. dependencies in continuous data, allowing flexible correlation structure learning.
• Derives a posterior distribution for merging times that follows a generalized inverse Gaussian (GIG) distribution, enabling efficient SMC resampling.
• Introduces a quadratic-time approximation (MPost2) that closely matches the performance of the exact method (MPost1) while being much faster.
• Derives a greedy algorithm as the mode of the posterior merging time distribution, which is shown to outperform the original greedy approach.

Experimental results

Research questions

• RQ1 Can Bayesian hierarchical clustering be effectively extended to non-i.i.d. continuous data using flexible priors?
• RQ2 Can the computational cost of Bayesian hierarchical clustering be reduced from O(n³) to O(n²) without sacrificing accuracy?
• RQ3 Does the proposed SMC sampler outperform existing inference methods in terms of speed and clustering quality?
• RQ4 Can a refined greedy algorithm derived from the posterior distribution improve upon the original greedy method, especially on small datasets?
• RQ5 How well do Gaussian process priors model correlation structures in continuous data within a hierarchical clustering framework?

Key findings

• The proposed SMC sampler achieves quadratic computational complexity, reducing runtime significantly compared to the original cubic-time method.
• The MPost2 approximation achieves nearly identical performance to MPost1 but with substantially improved computational efficiency.
• The refined greedy algorithm outperforms the original greedy method of Teh et al. (2008), particularly on small datasets, due to a more accurate posterior-based merging strategy.
• On the mocap dataset, the method produced a robust hierarchical structure in 50 iterations, with the best particle weight reaching 0.0784, and completed in under 10 minutes.
• The method demonstrated strong performance on both artificial and real-world data, with results showing improved clustering accuracy and structural fidelity.
• The use of Gaussian process priors enabled effective modeling of temporal and spatial correlations in continuous data, enhancing model flexibility and fit.