[Paper Review] Fast hierarchical clustering and other applications of dynamic closest pairs

David Eppstein|arXiv (Cornell University)|Jan 1, 1998

Complexity and Algorithms in Graphs34 references39 citations

TL;DR

This paper presents dynamic closest pair data structures for arbitrary distance functions, enabling O(n log n) amortized time per insertion/deletion using O(n) space, or optimal O(n) time with O(n²) space via a quadtree-like structure. The method accelerates hierarchical clustering, greedy matching, and TSP heuristics, outperforming prior heuristics in practice.

ABSTRACT

We develop data structures for dynamic closest pair problems with arbitrary (not necessarily geometric) distance functions, based on a technique previously used by the author for Euclidean closest pairs. We show how to insert and delete objects from an n-object set, maintaining the closest pair, in O(n log n) time per update and O(n) space. With quadratic space, we can instead use a quadtree-like structure to achieve an optimal time bound, O(n) per update. We apply these data structures to hierarchical clustering, greedy matching, and TSP heuristics, and discuss other potential applications in machine learning, Grobner bases, and local improvement algorithms for partition and placement problems. Experiments show our new methods to be faster in practice than previously used heuristics.

Motivation & Objective

• To develop efficient dynamic closest pair data structures that support insertions and deletions while maintaining the closest pair under arbitrary distance functions.
• To achieve optimal update time bounds—O(n) per update—using quadratic space via a quadtree-like structure.
• To apply the dynamic closest pair framework to hierarchical clustering, greedy matching, and TSP heuristics, improving practical performance.
• To explore broader applications in machine learning, Grobner bases, and local improvement algorithms for partition and placement problems.

Proposed method

• Leverages a dynamic closest pair technique previously applied to Euclidean distances, generalizing it to arbitrary distance functions.
• Employs a data structure that maintains the closest pair dynamically, supporting insertions and deletions in O(n log n) time with O(n) space.
• Uses a quadtree-like hierarchical decomposition to achieve O(n) update time per operation when using O(n²) space.
• Applies the dynamic closest pair structure to hierarchical clustering by incrementally merging closest pairs.
• Extends the framework to greedy matching and TSP heuristics by maintaining and updating the closest pair during construction.
• Designs a general-purpose interface for dynamic closest pairs that can be reused across diverse algorithmic applications.

Experimental results

Research questions

• RQ1 Can dynamic closest pair data structures be generalized beyond geometric distances to arbitrary distance functions?
• RQ2 What is the optimal time complexity achievable for dynamic closest pair updates using limited space?
• RQ3 How can dynamic closest pair data structures accelerate hierarchical clustering and other NP-hard heuristic problems?
• RQ4 Can the framework be applied to machine learning and algebraic computation tasks like Grobner bases?
• RQ5 What performance gains does the new method achieve over existing heuristics in practice?

Key findings

• The proposed method achieves O(n log n) amortized time per insertion or deletion with O(n) space, significantly improving upon previous approaches.
• With O(n²) space, the method achieves an optimal O(n) time per update, matching theoretical lower bounds.
• The dynamic closest pair framework accelerates hierarchical clustering, greedy matching, and TSP heuristics beyond prior heuristic methods.
• Experimental results demonstrate that the new algorithms are faster in practice than previously used heuristics.
• The framework is extensible to diverse domains, including machine learning, Grobner basis computation, and local improvement algorithms.