[Paper Review] Online Hierarchical Clustering Approximations

Aditya Krishna Menon, Anand Rajagopalan+4 more|arXiv (Cornell University)|Sep 20, 2019

Advanced Clustering Algorithms Research18 references4 citations

TL;DR

This paper proposes two online hierarchical clustering algorithms, OTD and OHAC, that approximate the Moseley-Wang (MW) revenue function for optimal hierarchical clustering. OTD uses a top-down approach with a $\frac{\beta}{3}$-approximation guarantee under data separation, while OHAC emulates offline HAC via a novel split-merge procedure, achieving near-optimal MW revenue and superior cluster quality with $O(n)$ per-round complexity.

ABSTRACT

Hierarchical clustering is a widely used approach for clustering datasets at multiple levels of granularity. Despite its popularity, existing algorithms such as hierarchical agglomerative clustering (HAC) are limited to the offline setting, and thus require the entire dataset to be available. This prohibits their use on large datasets commonly encountered in modern learning applications. In this paper, we consider hierarchical clustering in the online setting, where points arrive one at a time. We propose two algorithms that seek to optimize the Moseley and Wang (MW) revenue function, a variant of the Dasgupta cost. These algorithms offer different tradeoffs between efficiency and MW revenue performance. The first algorithm, OTD, is a highly efficient Online Top Down algorithm which provably achieves a 1/3-approximation to the MW revenue under a data separation assumption. The second algorithm, OHAC, is an online counterpart to offline HAC, which is known to yield a 1/3-approximation to the MW revenue, and produce good quality clusters in practice. We show that OHAC approximates offline HAC by leveraging a novel split-merge procedure. We empirically show that OTD and OHAC offer significant efficiency and cluster quality gains respectively over baselines.

Motivation & Objective

• To address the lack of efficient, theoretically grounded online hierarchical clustering algorithms for large-scale streaming data.
• To design online algorithms that optimize the Moseley and Wang (2017) revenue function, a principled objective for hierarchical clustering quality.
• To achieve a balance between computational efficiency and cluster quality in the online setting.
• To provide theoretical approximation guarantees and empirical validation against offline baselines and existing online methods.

Proposed method

• OTD is an online top-down hierarchical clustering algorithm that performs a single root-to-leaf tree search per incoming point, ensuring $O(d)$ per-round time complexity, where $d$ is the hierarchy depth.
• OTD achieves a $\frac{\beta}{3}$-approximation to the MW revenue, with $\beta$ quantifying data point separation, and a $\frac{1}{3}$-approximation under well-separated conditions.
• OHAC is an online counterpart to offline HAC, using a novel split-merge procedure to dynamically maintain cluster structure and approximate offline HAC's hierarchy.
• OHAC is shown to have $O(n)$ per-round time complexity for certain linkages and achieves MW revenue performance close to offline HAC.
• The algorithms are evaluated using MW revenue and triplet distance to measure approximation quality relative to offline HAC.
• Theoretical analysis includes proving approximation bounds under data separation assumptions, while empirical evaluation uses synthetic and real-world datasets including MNIST and ImageNet.

Experimental results

Research questions

• RQ1 Can online hierarchical clustering algorithms achieve theoretical approximation guarantees for the Moseley-Wang revenue function?
• RQ2 How does the performance of online algorithms compare to offline HAC in terms of cluster quality and runtime?
• RQ3 Can a top-down online algorithm like OTD achieve a strong approximation factor despite its simplicity and efficiency?
• RQ4 Does an online algorithm that emulates offline HAC via a split-merge mechanism preserve hierarchical structure and improve MW revenue?
• RQ5 What is the tradeoff between computational efficiency and cluster quality in online hierarchical clustering?

Key findings

• OTD consistently achieves or exceeds the theoretical $\frac{1}{3}$-approximation factor for MW revenue on real-world and synthetic datasets, even without strict data separation.
• OHAC achieves MW revenue performance nearly identical to offline HAC, validating its ability to closely approximate the optimal hierarchy.
• Offline HAC produces the highest cluster quality but is infeasible for large datasets like ImageNet, running out of memory at 70K points.
• OTD is significantly faster than both OHAC and offline HAC, with runtime growing sublinearly, making it suitable for high-throughput streaming applications.
• OHAC outperforms PERCH and other baselines in triplet distance, indicating better preservation of hierarchical structure compared to existing online methods.
• On the ImageNet dataset with over one million points, both OTD and OHAC complete within a reasonable time, while offline HAC fails to finish within 8 hours.