[Paper Review] From Trees to Continuous Embeddings and Back: Hyperbolic Hierarchical Clustering

Ines Chami, Albert Gu+2 more|arXiv (Cornell University)|Oct 1, 2020

Complex Network Analysis Techniques41 references32 citations

TL;DR

HypHC provides a differentiable hyperbolic relaxation of Dasgupta's hierarchical clustering, with a decoding step that yields a (1+ε)-approximate optimal tree and strong empirical results, plus end-to-end training potential.

ABSTRACT

Similarity-based Hierarchical Clustering (HC) is a classical unsupervised machine learning algorithm that has traditionally been solved with heuristic algorithms like Average-Linkage. Recently, Dasgupta reframed HC as a discrete optimization problem by introducing a global cost function measuring the quality of a given tree. In this work, we provide the first continuous relaxation of Dasgupta's discrete optimization problem with provable quality guarantees. The key idea of our method, HypHC, is showing a direct correspondence from discrete trees to continuous representations (via the hyperbolic embeddings of their leaf nodes) and back (via a decoding algorithm that maps leaf embeddings to a dendrogram), allowing us to search the space of discrete binary trees with continuous optimization. Building on analogies between trees and hyperbolic space, we derive a continuous analogue for the notion of lowest common ancestor, which leads to a continuous relaxation of Dasgupta's discrete objective. We can show that after decoding, the global minimizer of our continuous relaxation yields a discrete tree with a (1 + epsilon)-factor approximation for Dasgupta's optimal tree, where epsilon can be made arbitrarily small and controls optimization challenges. We experimentally evaluate HypHC on a variety of HC benchmarks and find that even approximate solutions found with gradient descent have superior clustering quality than agglomerative heuristics or other gradient based algorithms. Finally, we highlight the flexibility of HypHC using end-to-end training in a downstream classification task.

Motivation & Objective

• Motivate the need for a global objective in Hierarchical Clustering (HC) and provide a continuous relaxation of Dasgupta’s discrete HC cost.
• Introduce a leaf-hyperbolic embedding representation that encodes tree structure without explicit internal-node parameters.
• Derive a differentiable Hyperbolic LCA analogue and a corresponding continuous cost for HC.
• Provide a decoding procedure to recover a discrete dendrogram from continuous embeddings with approximation guarantees.
• Demonstrate empirical gains over agglomerative and gradient-based baselines and explore end-to-end classification integration.

Proposed method

• Represent binary trees via leaf embeddings in the Poincaré disk (hyperbolic space).
• Define a continuous hyperbolic LCA as the point on the geodesic between two leaves closest to the origin, enabling a differentiable Dasgupta-like objective.
• Use a scaled softmax over hyperbolic LCA depths to form the HypHC differentiable cost C_HypHC(Z; w, τ).
• Decode embeddings into a binary tree with a procedure that iteratively merges the most similar pair by their hyperbolic LCA distance to the origin (Algorithm 1).
• Prove that with spread embeddings, solving the continuous relaxation yields a (1+ε)-approximation to the discrete Dasgupta optimum (τ constrained to O(1/log(1/ε))).
• Offer practical optimizations: triplet sampling and greedy decoding to scale to large datasets, plus end-to-end training strategies for downstream tasks.

Experimental results

Research questions

• RQ1 Can a differentiable continuous relaxation of Dasgupta’s HC objective achieve comparable or better clustering quality than traditional HC heuristics?
• RQ2 How can hyperbolic leaf embeddings capture the underlying tree structure and allow decoding back to a dendrogram with low distortion?
• RQ3 What are the theoretical guarantees linking the continuous relaxation to the discrete optimal HC, and under what conditions (e.g., spread embeddings) do they hold?
• RQ4 Do gradient-based HypHC solutions outperform classical agglomerative methods and prior gradient-based HC approaches on standard benchmarks?
• RQ5 Can HypHC be effectively integrated into end-to-end ML pipelines, including downstream classification tasks?

Key findings

• A continuous HypHC objective with a hyperbolic LCA yields a (1+ε)-approximation to Dasgupta’s optimal tree when embeddings are spread and τ is appropriately chosen.
• Approximate gradient-based solutions outperform or match discrete and other continuous HC methods on multiple benchmarks.
• Greedy decoding and triplet sampling provide substantial speedups (e.g., orders of magnitude) with minimal loss in clustering quality.
• HypHC embeddings can be trained end-to-end with downstream classification, yielding improved accuracy over two-stage embed-then-classify baselines.
• Empirical results show HypHC surpasses or matches best discrete methods and outperforms the prior continuous approach UFit across several datasets.