[Paper Review] A tight lower bound instance for k-means++ in constant dimension

Anup Bhattacharya, Ragesh Jaiswal+1 more|arXiv (Cornell University)|Jan 13, 2014

Data Management and Algorithms13 references4 citations

TL;DR

This paper presents a two-dimensional dataset that demonstrates a tight lower bound for the k-means++ seeding algorithm, showing it achieves an $ O(\log k) $ approximation ratio with probability exponentially small in $ k $. The construction resolves open problems on the algorithm's performance in low dimensions by proving that even in constant dimension, k-means++ can fail with high probability despite its expected $ O(\log k) $ approximation guarantee.

ABSTRACT

The k-means++ seeding algorithm is one of the most popular algorithms that is used for finding the initial $k$ centers when using the k-means heuristic. The algorithm is a simple sampling procedure and can be described as follows: Pick the first center randomly from the given points. For $i > 1$, pick a point to be the $i^{th}$ center with probability proportional to the square of the Euclidean distance of this point to the closest previously $(i-1)$ chosen centers. The k-means++ seeding algorithm is not only simple and fast but also gives an $O(\log{k})$ approximation in expectation as shown by Arthur and Vassilvitskii. There are datasets on which this seeding algorithm gives an approximation factor of $Ω(\log{k})$ in expectation. However, it is not clear from these results if the algorithm achieves good approximation factor with reasonably high probability (say $1/poly(k)$). Brunsch and Röglin gave a dataset where the k-means++ seeding algorithm achieves an $O(\log{k})$ approximation ratio with probability that is exponentially small in $k$. However, this and all other known lower-bound examples are high dimensional. So, an open problem was to understand the behavior of the algorithm on low dimensional datasets. In this work, we give a simple two dimensional dataset on which the seeding algorithm achieves an $O(\log{k})$ approximation ratio with probability exponentially small in $k$. This solves open problems posed by Mahajan et al. and by Brunsch and Röglin.

Motivation & Objective

• To close the open problem of whether k-means++ can achieve poor approximation with high probability in low-dimensional datasets.
• To provide a tight lower bound instance for k-means++ in constant dimension, specifically in 2D.
• To show that the algorithm's expected $ O(\log k) $ approximation guarantee does not imply good performance with constant probability in low dimensions.
• To resolve a long-standing open question posed by Mahajan et al. and Brunsch and Röglin regarding the behavior of k-means++ in planar (2D) settings.

Proposed method

• Construct a carefully designed two-dimensional dataset with clustered points and a single outlier to simulate worst-case behavior.
• Analyze the k-means++ seeding process using a Markov chain model to track the probability of selecting distant centers.
• Use Hoeffding's inequality to bound the tail probability that the algorithm fails to select a good center set within $ k-1 $ steps.
• Establish a lower bound on the expected number of steps required to reach a good configuration, showing it grows faster than polynomial in $ k $.
• Leverage probabilistic concentration bounds and asymptotic analysis to show that the success probability decays exponentially in $ k $.
• Prove that the algorithm's approximation ratio is at most $ \delta \cdot \log k $ with probability at most $ \exp(-(k-1)^{1-o(1)-120\delta}) $, which is exponentially small.

Experimental results

Research questions

• RQ1 Can k-means++ achieve a poor approximation ratio with high probability in low-dimensional datasets, despite its expected $ O(\log k) $ guarantee?
• RQ2 Is there a two-dimensional dataset where k-means++ fails to achieve a good approximation with probability exponentially small in $ k $?
• RQ3 Does the algorithm's performance degrade significantly in constant dimension, even when the expected approximation is bounded?
• RQ4 Can a tight lower bound instance for k-means++ be constructed in 2D, resolving prior open problems in the literature?
• RQ5 What is the exact trade-off between the number of centers and the success probability of k-means++ in low-dimensional spaces?

Key findings

• The paper constructs a two-dimensional dataset where k-means++ achieves an $ O(\log k) $ approximation ratio with probability at most $ \exp(-(k-1)^{1-o(1)-120\delta}) $, which is exponentially small in $ k $.
• This resolves an open problem by Mahajan et al. and Brunsch and Röglin, showing that k-means++ can fail with high probability even in constant dimension.
• The failure probability is not just in expectation but occurs with exponentially small success probability, demonstrating a strong lower bound.
• The analysis shows that the expected $ O(\log k) $ approximation guarantee does not imply high-probability performance in low dimensions.
• The constructed instance is tight, meaning the $ O(\log k) $ approximation factor is best possible in expectation, and this bound is matched by a failure probability that is exponentially small in $ k $.
• The result proves that the algorithm's performance in low dimensions is fundamentally limited by the same worst-case behavior observed in high dimensions, but now in a 2D setting.