[Paper Review] Efficient Active Algorithms for Hierarchical Clustering

Akshay Krishnamurthy, Sivaraman Balakrishnan+2 more|arXiv (Cornell University)|Jun 18, 2012

Advanced Clustering Algorithms Research13 references27 citations

TL;DR

This paper proposes a general active learning framework for hierarchical clustering that drastically reduces the number of similarity measurements required by iteratively clustering small, randomly sampled subsets of data. The method achieves theoretical guarantees, recovering clusters of size Ω(log n) using O(n log²n) similarities and running in O(n log³n) time, with empirical validation showing significant speedups and strong clustering performance on real datasets.

ABSTRACT

Advances in sensing technologies and the growth of the internet have resulted in an explosion in the size of modern datasets, while storage and processing power continue to lag behind. This motivates the need for algorithms that are efficient, both in terms of the number of measurements needed and running time. To combat the challenges associated with large datasets, we propose a general framework for active hierarchical clustering that repeatedly runs an off-the-shelf clustering algorithm on small subsets of the data and comes with guarantees on performance, measurement complexity and runtime complexity. We instantiate this framework with a simple spectral clustering algorithm and provide concrete results on its performance, showing that, under some assumptions, this algorithm recovers all clusters of size ?(log n) using O(n log^2 n) similarities and runs in O(n log^3 n) time for a dataset of n objects. Through extensive experimentation we also demonstrate that this framework is practically alluring.

Motivation & Objective

• To address the computational and measurement burden of large-scale hierarchical clustering by minimizing the number of pairwise similarity computations.
• To develop a general framework that can be applied to off-the-shelf clustering algorithms, enabling active, measurement-efficient clustering.
• To provide theoretical guarantees on cluster recovery, measurement complexity, and runtime for active hierarchical clustering.
• To demonstrate practical efficiency and accuracy through extensive experiments on real-world and synthetic datasets.

Proposed method

• The framework uses a recursive active clustering strategy: at each level, it samples a small subset of size s from the current dataset and applies a base clustering algorithm (e.g., spectral clustering) to this subset.
• The algorithm leverages statistical guarantees from prior work (Balakrishnan et al., 2011) to ensure that the clustering of the small subset reflects the structure of the full dataset under mild assumptions.
• It employs a hierarchical approach where clusters are refined iteratively, with each level using a new round of active sampling and clustering on the current cluster set.
• The method is instantiated with spectral clustering by computing eigenvectors only on small sub-matrices of the similarity matrix, avoiding full spectral decomposition.
• The framework allows tuning of the sampling size s to balance measurement overhead, computational cost, and statistical accuracy.
• It includes a pruning step to remove small clusters that bias performance metrics, focusing on clusters of size Ω(log n).

Experimental results

Research questions

• RQ1 Can a general active learning framework be designed for hierarchical clustering that reduces the number of similarity measurements while preserving clustering accuracy?
• RQ2 What theoretical guarantees can be provided for cluster recovery, measurement complexity, and runtime in such an active framework?
• RQ3 How does the performance of active spectral clustering compare to standard spectral and k-means clustering in terms of accuracy and efficiency?
• RQ4 Can the framework be applied effectively to real-world datasets with complex structures, such as biological sequences or network topologies?

Key findings

• The ActiveSpectral algorithm recovers all clusters of size Ω(log n) with high probability using O(n log²n) similarity measurements and O(n log³n) runtime for a dataset of size n.
• On real-world datasets like SNP and phylogeny, the active algorithms (ActiveSpec and ActiveKMeans) achieved significant speedups—running in under 20 seconds compared to over 130 seconds for standard spectral clustering—while maintaining high clustering quality.
• The active algorithms achieved outlier fractions of 0.019 (ActiveSpec) and 0.018 (ActiveKMeans) on the SNP dataset, outperforming non-active baselines in terms of agreement with reference hierarchies.
• Heatmaps of permuted similarity matrices showed clear block structures for ActiveSpectral and ActiveKMeans on SNP and phylogeny datasets, indicating strong clustering performance.
• The framework demonstrated robustness on NIPS and RTW datasets, though performance degraded on RTW due to the presence of many small, undersampled clusters.
• The results suggest that active algorithms can efficiently recover high-rank matrices (e.g., rank n/log n) using O(n log²n) similarities, offering potential for matrix completion applications.