[Paper Review] Uniform Concentration Bounds toward a Unified Framework for Robust Clustering

Debolina Paul, Saptarshi Chakraborty+2 more|arXiv (Cornell University)|Oct 26, 2021

Statistical Methods and Inference4 citations

TL;DR

This paper proposes a unified, robust framework for center-based clustering using Median-of-Means (MoM) estimation under a general class of Bregman divergence losses. It establishes uniform concentration bounds via Rademacher complexities and Dudley’s chaining, achieving strong consistency and an $O(n^{-1/2})$ error rate without distributional assumptions on outliers or restrictions on the $n$-to-$p$ ratio, subsuming and theoretically strengthening existing $k$-means variants.

ABSTRACT

Recent advances in center-based clustering continue to improve upon the drawbacks of Lloyd's celebrated $k$-means algorithm over $60$ years after its introduction. Various methods seek to address poor local minima, sensitivity to outliers, and data that are not well-suited to Euclidean measures of fit, but many are supported largely empirically. Moreover, combining such approaches in a piecemeal manner can result in ad hoc methods, and the limited theoretical results supporting each individual contribution may no longer hold. Toward addressing these issues in a principled way, this paper proposes a cohesive robust framework for center-based clustering under a general class of dissimilarity measures. In particular, we present a rigorous theoretical treatment within a Median-of-Means (MoM) estimation framework, showing that it subsumes several popular $k$-means variants. In addition to unifying existing methods, we derive uniform concentration bounds that complete their analyses, and bridge these results to the MoM framework via Dudley's chaining arguments. Importantly, we neither require any assumptions on the distribution of the outlying observations nor on the relative number of observations $n$ to features $p$. We establish strong consistency and an error rate of $O(n^{-1/2})$ under mild conditions, surpassing the best-known results in the literature. The methods are empirically validated thoroughly on real and synthetic datasets.

Motivation & Objective

• To address the lack of finite-sample theoretical guarantees in robust $k$-means variants that are empirically successful but theoretically fragile.
• To unify diverse center-based clustering methods—such as $k$-means, $k$-medians, and Bregman $k$-means—under a single robust framework based on Median-of-Means (MoM) estimation.
• To provide rigorous statistical analysis, including uniform concentration bounds, for MoM-based clustering under general Bregman divergences without assuming i.i.d. or light-tailed distributions for outliers.
• To establish strong consistency and finite-sample error rates that surpass existing results, particularly under minimal assumptions on the relationship between sample size $n$ and dimension $p$.
• To bridge theoretical robustness with practical performance by demonstrating stability across varying cluster counts and outlier levels in empirical evaluations.

Proposed method

• The framework employs a Median-of-Means (MoM) estimation strategy to robustify center-based clustering, dividing data into blocks and minimizing the median of block-wise empirical risk.
• It generalizes to any Bregman divergence loss, not just squared Euclidean distance, enabling application to exponential family models.
• Uniform concentration bounds are derived using Rademacher complexities and Dudley’s chaining arguments, providing finite-sample control over estimation error.
• The analysis assumes only i.i.d. sampling for inliers, with no restrictions on the outlier set—outliers may be arbitrary, heavy-tailed, or dependent.
• The method maintains per-iteration complexity comparable to Lloyd’s $k$-means, enabling efficient optimization via gradient-based algorithms.
• Theoretical guarantees are established under mild regularity conditions, including boundedness of cluster centroids and sub-Gaussian-like behavior of inlier distributions.

Experimental results

Research questions

• RQ1 Can a unified theoretical framework be developed for robust center-based clustering that subsumes existing $k$-means variants under general dissimilarity measures?
• RQ2 Can uniform concentration bounds be established for MoM-based clustering objectives without assuming light-tailed or i.i.d. outliers?
• RQ3 What finite-sample error rates can be achieved under minimal assumptions, particularly when $n$ and $p$ are not in a restrictive relationship?
• RQ4 How does the MoM framework improve robustness to outliers compared to empirical risk minimization (ERM) approaches in practice?
• RQ5 Can the theoretical error rate $O(n^{-1/2})$ be achieved without asymptotic or distributional assumptions on outliers?

Key findings

• The proposed MoM-based clustering framework achieves strong consistency and an $O(n^{-1/2})$ error rate under mild conditions, improving upon the best-known results in the literature.
• The framework unifies multiple $k$-means variants, including $k$-means, $k$-medians, and Bregman $k$-means, under a single theoretical umbrella.
• Uniform concentration bounds are established via Rademacher complexities and Dudley’s chaining, enabling finite-sample analysis without requiring moment conditions on outliers.
• Empirical results show that MOMPKM (MoM Power $k$-means) maintains stable performance across increasing cluster counts and outlier percentages, outperforming ERM-based and non-robust methods.
• The error rate scales as $O(n^{(eta-1)/2})$ when the number of outliers is $O(n^{eta})$ for $0 < eta < 1$, with consistency requiring $| olimits ext{outliers}| = o(n)$.
• The method remains robust even when outliers are unbounded or dependent, demonstrating that MoM estimation effectively mitigates their impact without distributional assumptions.