[Paper Review] Recovery guarantees for exemplar-based clustering

Abhinav Nellore, Rachel Ward|arXiv (Cornell University)|Sep 12, 2013

Sparse and Compressive Sensing Techniques53 references4 citations

TL;DR

This paper establishes theoretical recovery guarantees for exemplar-based clustering using a linear programming (LP) relaxation of the k-medoids problem. It proves that under certain isotropic ball distributions in d ≥ 2 dimensions, the LP relaxation exactly recovers the true k-medoid solution with high probability when the number of points and inter-cluster separation are sufficiently large—even in regimes where standard distance-thresholding fails.

ABSTRACT

For a certain class of distributions, we prove that the linear programming relaxation of $k$-medoids clustering---a variant of $k$-means clustering where means are replaced by exemplars from within the dataset---distinguishes points drawn from nonoverlapping balls with high probability once the number of points drawn and the separation distance between any two balls are sufficiently large. Our results hold in the nontrivial regime where the separation distance is small enough that points drawn from different balls may be closer to each other than points drawn from the same ball; in this case, clustering by thresholding pairwise distances between points can fail. We also exhibit numerical evidence of high-probability recovery in a substantially more permissive regime.

Motivation & Objective

• To establish theoretical conditions under which the LP relaxation of k-medoids clustering exactly recovers the true globally optimal solution.
• To analyze recovery performance in a nontrivial regime where inter-cluster distances are smaller than some intra-cluster distances, rendering thresholding-based methods ineffective.
• To extend prior results beyond one-dimensional settings to higher-dimensional Euclidean spaces (d ≥ 2) with isotropic cluster structures.
• To investigate the robustness of LP relaxation to small separation distances and low sample sizes per cluster.
• To explore the gap between theoretical guarantees and empirical success in simulations, particularly for non-uniform and high-dimensional distributions.

Proposed method

• Formulates k-medoids clustering as a binary integer program minimizing total squared distance to assigned medoids.
• Relaxes the binary constraints to nonnegativity, yielding a linear program (LinKMed) that can be solved efficiently.
• Analyzes the LP relaxation under a generative model where points are i.i.d. drawn from k non-overlapping, isotropic d-dimensional balls.
• Uses probabilistic arguments and geometric concentration to prove that the optimal solution to the LP matches the integer solution with high probability.
• Employs KKT conditions and duality to analyze optimality and uniqueness of the solution under the assumed model.
• Conducts numerical simulations across varying dimensions d, separation radii R, and sample sizes n to evaluate recovery performance beyond theoretical bounds.

Experimental results

Research questions

• RQ1 Under what conditions does the LP relaxation of k-medoids exactly recover the true k-medoid solution with high probability?
• RQ2 How does the recovery performance of the LP relaxation compare to thresholding-based clustering in regimes where intra-cluster distances exceed inter-cluster distances?
• RQ3 Can theoretical recovery guarantees be extended to higher-dimensional spaces (d ≥ 2) and non-uniform distributions within clusters?
• RQ4 How does the number of points per cluster and inter-cluster separation affect the likelihood of exact recovery via LP relaxation?
• RQ5 To what extent do empirical results exceed the theoretical bounds, particularly in high-dimensional or non-isotropic settings?

Key findings

• The LP relaxation of k-medoids exactly recovers the true k-medoid solution with high probability when points are drawn from k non-overlapping d-dimensional balls (d ≥ 2), provided the number of points and inter-ball separation are sufficiently large.
• Recovery is guaranteed even in a nontrivial regime where some points from different clusters are closer than some points from the same cluster, a regime where simple distance-thresholding fails.
• Numerical simulations show high-probability recovery for uniform distributions in d=2 even at R ≈ 2.5 and n in the tens, significantly outperforming the theoretical bound requiring n ≥ 10^6 for R=3.75.
• For fixed n, R, and d, recovery improves as the number of clusters k decreases, suggesting that recovery probability decreases with increasing k.
• As dimension d increases, the number of successful ball recoveries increases, even under uniform distributions, indicating that higher dimensions enhance the separability of clusters under the LP relaxation.
• Theoretical guarantees are conservative compared to empirical performance, especially in high dimensions or with non-uniform distributions concentrating mass toward cluster centers.