[Paper Review] Modern hierarchical, agglomerative clustering algorithms

Daniel Müllner|arXiv (Cornell University)|Sep 12, 2011

Advanced Clustering Algorithms Research369 citations

TL;DR

This paper introduces a new generic agglomerative clustering algorithm that efficiently handles any distance update scheme, significantly outperforming existing methods for centroid and median linkage. It proves the correctness of Rohlf’s and Murtagh’s algorithms, establishes optimal algorithm recommendations for each linkage scheme, and provides implementations for R and Python, bridging theory and practice in standard clustering software.

ABSTRACT

This paper presents algorithms for hierarchical, agglomerative clustering which perform most efficiently in the general-purpose setup that is given in modern standard software. Requirements are: (1) the input data is given by pairwise dissimilarities between data points, but extensions to vector data are also discussed (2) the output is a "stepwise dendrogram", a data structure which is shared by all implementations in current standard software. We present algorithms (old and new) which perform clustering in this setting efficiently, both in an asymptotic worst-case analysis and from a practical point of view. The main contributions of this paper are: (1) We present a new algorithm which is suitable for any distance update scheme and performs significantly better than the existing algorithms. (2) We prove the correctness of two algorithms by Rohlf and Murtagh, which is necessary in each case for different reasons. (3) We give well-founded recommendations for the best current algorithms for the various agglomerative clustering schemes.

Motivation & Objective

• Address the gap between theoretical advances in hierarchical clustering and suboptimal implementations in standard software.
• Develop efficient, correct, and practical algorithms for agglomerative clustering under the common input/output format used in modern statistical software.
• Provide formal correctness proofs for previously unproven algorithms, including Rohlf’s MST-based single linkage and Murtagh’s nearest-neighbor chain algorithm.
• Recommend the best-performing algorithm for each of the seven major agglomerative clustering schemes based on theoretical and empirical analysis.
• Extend the scope to vector data by suggesting efficient adaptations for Euclidean distance-based schemes like Ward, centroid, and median linkage.

Proposed method

• Proposes a new generic clustering algorithm (Generic_linkage and its variant) that dynamically maintains nearest neighbor candidates and updates inter-cluster distances using a priority queue and in-place array updates.
• Adapts the minimum spanning tree (MST) algorithm for single linkage clustering, enabling on-the-fly distance computation and reducing memory overhead.
• Applies Murtagh’s nearest-neighbor chain algorithm to complete, average, weighted, and Ward linkage schemes, using centroid-based distance updates.
• Uses a stepwise dendrogram as the output data structure, ensuring compatibility with standard software like R and SciPy.
• Implements the algorithms in C++ with interfaces to R and Python, enabling practical deployment.
• Employs a dynamic data structure where each cluster is represented by a label, and nearest neighbors are tracked via a priority queue (mindist) to avoid redundant distance computations.

Experimental results

Research questions

• RQ1 Which agglomerative clustering algorithm performs optimally across all standard linkage schemes under the general-purpose input/output format?
• RQ2 Why are existing software implementations suboptimal, and how can theoretical improvements be effectively bridged to practical use?
• RQ3 What are the correctness conditions for Rohlf’s MST-based algorithm and Murtagh’s nearest-neighbor chain algorithm, and why were their proofs previously missing?
• RQ4 How does the choice of output data structure (e.g., stepwise dendrogram) affect algorithm validity and efficiency?
• RQ5 Can a single generic algorithm be designed to support arbitrary distance update formulas while maintaining high performance?

Key findings

• The proposed Generic_linkage algorithm outperforms existing methods for centroid and median linkage schemes, with a simpler variant (Generic_linkage_variant) showing reduced recalculations in practice.
• The correctness of Rohlf’s MST-based algorithm for single linkage and Murtagh’s nearest-neighbor chain algorithm for complete/average/weighted/Ward linkage is formally proven for the first time.
• The MST algorithm enables on-the-fly distance computation, eliminating the need to precompute the full dissimilarity matrix and reducing memory usage.
• For Euclidean vector data, the Generic_linkage algorithm and its variant are optimal for Ward, centroid, and median linkage, with performance gains due to efficient nearest-neighbor tracking.
• The stepwise dendrogram is shown to be a necessary and sufficient output structure for ensuring algorithm correctness and compatibility with standard software.
• The paper establishes that no algorithm can achieve better than Ω(N²) time complexity in the general dissimilarity input setting, confirming the theoretical lower bound.