[Paper Review] A simple D^2-sampling based PTAS for k-means and other Clustering Problems

Ragesh Jaiswal, Amit Kumar+1 more|arXiv (Cornell University)|Jan 20, 2012

Sparse and Compressive Sensing Techniques19 references4 citations

TL;DR

This paper presents a simple randomized $(1+\epsilon)$-approximation algorithm for the $k$-means problem using $D^2$-sampling as its core mechanism. By iteratively sampling points with $D^2$-sampling and selecting centroids from these samples, the algorithm achieves a $(1+\epsilon)$-approximation in $O(nd \cdot 2^{\tilde{O}(k^2/\epsilon)})$ time, offering a clean alternative to complex coreset-based PTAS approaches.

ABSTRACT

Given a set of points $P \subset \mathbb{R}^d$, the $k$-means clustering problem is to find a set of $k$ {\em centers} $C = \{c_1,...,c_k\}, c_i \in \mathbb{R}^d,$ such that the objective function $\sum_{x \in P} d(x,C)^2$, where $d(x,C)$ denotes the distance between $x$ and the closest center in $C$, is minimized. This is one of the most prominent objective functions that have been studied with respect to clustering. $D^2$-sampling \cite{ArthurV07} is a simple non-uniform sampling technique for choosing points from a set of points. It works as follows: given a set of points $P \subseteq \mathbb{R}^d$, the first point is chosen uniformly at random from $P$. Subsequently, a point from $P$ is chosen as the next sample with probability proportional to the square of the distance of this point to the nearest previously sampled points. $D^2$-sampling has been shown to have nice properties with respect to the $k$-means clustering problem. Arthur and Vassilvitskii \cite{ArthurV07} show that $k$ points chosen as centers from $P$ using $D^2$-sampling gives an $O(\log{k})$ approximation in expectation. Ailon et. al. \cite{AJMonteleoni09} and Aggarwal et. al. \cite{AggarwalDK09} extended results of \cite{ArthurV07} to show that $O(k)$ points chosen as centers using $D^2$-sampling give $O(1)$ approximation to the $k$-means objective function with high probability. In this paper, we further demonstrate the power of $D^2$-sampling by giving a simple randomized $(1 + ε)$-approximation algorithm that uses the $D^2$-sampling in its core.

Motivation & Objective

• To develop a simple, efficient $(1+\epsilon)$-approximation algorithm for the $k$-means clustering problem.
• To demonstrate that $D^2$-sampling alone, without complex coreset constructions, can yield a PTAS for $k$-means and related clustering objectives.
• To extend the applicability of $D^2$-sampling to other distance measures such as Mahalanobis distance and $\mu$-similar Bregman divergences.
• To provide a runtime-efficient algorithm with $O(nd \cdot 2^{\tilde{O}(k^2/\epsilon)})$ time complexity for constant $k$, simplifying prior approaches.

Proposed method

• The algorithm uses $D^2$-sampling to iteratively select a small set of candidate points based on their distance to already chosen centers.
• From each sampled set, the algorithm selects the centroid of a subset as the next center, ensuring diversity and representativeness.
• The process is repeated until $k$ centers are selected, with high probability yielding a $(1+\epsilon)$-approximate solution.
• The method relies on the $(f,\gamma,\delta)$-sampling property and centroid stability under $D^2$-sampling to ensure approximation guarantees.
• Theoretical analysis leverages symmetry, triangle inequality, and centroid properties of the distance function to bound approximation error.
• The algorithm is extended to Mahalanobis distance and $\mu$-similar Bregman divergences by verifying their compatibility with the required properties.

Experimental results

Research questions

• RQ1 Can $D^2$-sampling alone be used to design a simple PTAS for the $k$-means problem without relying on coreset constructions?
• RQ2 What is the runtime complexity of a $D^2$-sampling-based algorithm that achieves a $(1+\epsilon)$-approximation for $k$-means?
• RQ3 Do $D^2$-sampling and centroid selection generalize to other distance measures like Mahalanobis distance and $\mu$-similar Bregman divergences?
• RQ4 How does the approximation factor depend on the parameters $k$ and $\epsilon$ in the proposed algorithm?

Key findings

• The proposed algorithm achieves a $(1+\epsilon)$-approximation for the $k$-means objective using only $D^2$-sampling and centroid selection.
• The algorithm runs in $O(nd \cdot 2^{\tilde{O}(k^2/\epsilon)})$ time, which is linear in $n$ and $d$ for constant $k$.
• For Mahalanobis distance, the algorithm provides a $(1+\epsilon)$-approximation in $O(nd \cdot 2^{\tilde{O}(k^2/\epsilon)})$ time.
• For $\mu$-similar Bregman divergences, the algorithm achieves a $(1+\epsilon)$-approximation in $O(nd \cdot 2^{\tilde{O}(k^2/(\mu\epsilon))})$ time.
• The method avoids complex coreset constructions, offering a simpler alternative to prior PTAS approaches for $k$-means.