[Paper Review] Hierarchical Clustering Given Confidence Intervals of Metric Distances

Weiyu Huang, Alejandro Ribeiro|arXiv (Cornell University)|Oct 13, 2016

Topological and Geometric Data Analysis15 references4 citations

TL;DR

This paper proposes an axiomatic framework for hierarchical clustering when metric distances are known only within confidence intervals. It introduces two extremal methods—combine-and-cluster and cluster-and-combine—that satisfy axioms of value and transformation, providing universal upper and lower bounds on all admissible clustering methods, with demonstrated utility in tracking moving points and differentiating brain connectivity networks.

ABSTRACT

This paper considers metric spaces where distances between a pair of nodes are represented by distance intervals. The goal is to study methods for the determination of hierarchical clusters, i.e., a family of nested partitions indexed by a resolution parameter, induced from the given distance intervals of the metric spaces. Our construction of hierarchical clustering methods is based on defining admissible methods to be those methods that abide to the axioms of value - nodes in a metric space with two nodes are clustered together at the convex combination of the distance bounds between them - and transformation - when both distance bounds are reduced, the output may become more clustered but not less. Two admissible methods are constructed and are shown to provide universal upper and lower bounds in the space of admissible methods. Practical implications are explored by clustering moving points via snapshots and by clustering networks representing brain structural connectivity using the lower and upper bounds of the network distance. The proposed clustering methods succeed in identifying underlying clustering structures via the maximum and minimum distances in all snapshots, as well as in differentiating brain connectivity networks of patients from those of healthy controls.

Motivation & Objective

• To develop a formal axiomatic approach for hierarchical clustering when exact distances are unknown but bounded by confidence intervals.
• To define admissible clustering methods that satisfy the Axiom of Value (clustering at convex combinations of bounds) and Axiom of Transformation (monotonicity under uniform dominance).
• To construct two extremal methods—combine-and-cluster and cluster-and-combine—that provide universal upper and lower bounds across all admissible methods.
• To demonstrate practical utility in clustering moving points via snapshots and in differentiating brain structural connectivity networks of patients and healthy controls.

Proposed method

• The combine-and-cluster method estimates pairwise distances as a convex combination of lower and upper bounds, then applies single linkage clustering using the maximum edge weight in a chain as the cost.
• The cluster-and-combine method computes separate single linkage hierarchies on the lower and upper bound distance matrices, then combines them via the same convex combination to form the final ultrametric.
• Axiom of Value is enforced by requiring that two-node systems cluster precisely at the convex combination of their distance bounds, parameterized by a confidence weight α.
• Axiom of Transformation is adapted to require that if one network’s bounds uniformly dominate another’s, its resulting clusters must be at least as refined (i.e., more clustered).
• Theoretical analysis proves that combine-and-cluster yields the coarsest possible clustering (upper bound), while cluster-and-combine yields the finest (lower bound), across all admissible methods.
• The framework uses α-distance-reducing maps to prove that the minimum separation in the resulting ultrametric is bounded below by the convex combination of interval extremes.

Experimental results

Research questions

• RQ1 How can hierarchical clustering be consistently defined when only confidence intervals for metric distances are available?
• RQ2 What axioms ensure that clustering behavior remains logically coherent under uncertainty in distances?
• RQ3 Can extremal clustering methods be constructed that bound all other admissible methods under these axioms?
• RQ4 How do the proposed methods perform in real-world applications with uncertain or interval-based distance data?
• RQ5 Can the methods differentiate meaningful structural differences in dynamic or biological networks, such as brain connectivity?

Key findings

• The combine-and-cluster method produces the coarsest possible clustering among all admissible methods, serving as a universal upper bound.
• The cluster-and-combine method produces the finest possible clustering among all admissible methods, serving as a universal lower bound.
• Both methods satisfy the Axiom of Value and the adapted Axiom of Transformation, ensuring logical consistency under uncertainty.
• In the moving points experiment, the methods successfully identified underlying clustering structures using only the maximum and minimum distances across snapshots.
• In brain network analysis, the methods differentiated connectivity patterns between patients and healthy controls, with the lower and upper bound methods revealing distinct clustering behaviors.
• The theoretical framework proves that any admissible method must yield an ultrametric whose pairwise separations are bounded between the convex combination of interval extremes and the results of the two proposed extremal methods.