[Paper Review] Generalizing k-means for an arbitrary distance matrix

Balázs Szalkai|arXiv (Cornell University)|Mar 24, 2013

Data Management and Algorithms4 references3 citations

TL;DR

This paper generalizes k-means clustering to work with arbitrary distance matrices, even when data points lack a Euclidean representation. By expressing centroid distances as quadratic forms over the squared distance matrix, the method enables k-means to operate without vector inputs, extending its applicability to non-Euclidean and abstract data types.

ABSTRACT

The original k-means clustering method works only if the exact vectors representing the data points are known. Therefore calculating the distances from the centroids needs vector operations, since the average of abstract data points is undefined. Existing algorithms can be extended for those cases when the sole input is the distance matrix, and the exact representing vectors are unknown. This extension may be named relational k-means after a notation for a similar algorithm invented for fuzzy clustering. A method is then proposed for generalizing k-means for scenarios when the data points have absolutely no connection with a Euclidean space.

Motivation & Objective

• Address the limitation of standard k-means, which requires data points to be in a Euclidean space for centroid computation.
• Enable k-means clustering when only a distance matrix is available, with no access to underlying vector representations.
• Generalize k-means to work with arbitrary (potentially non-metric) distance matrices by redefining centroid distances via quadratic forms.
• Ensure the generalized method remains computationally feasible despite increased complexity from matrix operations.
• Investigate conditions under which the generalized method produces meaningful, non-negative distances to maintain interpretability.

Proposed method

• Represent data points via a squared distance matrix $ A \in \mathbb{R}^{n \times n} $, where $ A_{ij} = ||p_i - p_j||^2 $, even when $ p_i $ are not explicitly known.
• Express the squared distance from a point $ p_i $ to its cluster centroid $ z_i $ as a quadratic form: $ ||p_i - z_i||^2 = -\frac{1}{2} \lambda^T A \lambda $, where $ \lambda = \frac{1}{|S|}\chi(S) - e_i $ and $ \sum \lambda_j = 0 $.
• Use the property that $ \sum \lambda_j = 0 $ to derive the quadratic form expression, which depends only on the distance matrix $ A $, not on the original vectors.
• Apply the generalized distance formula even when $ A $ is not derived from a Euclidean space, enabling relational k-means on abstract data.
• Address potential negative distances by ensuring the quadratic form is non-negative, using matrix correction techniques like $ \beta $-spread transformation.
• Modify the distance matrix minimally to make the quadratic form negative semi-definite on the hyperplane $ \sum \lambda_j = 0 $, preserving original distances as much as possible.

Experimental results

Research questions

• RQ1 Can k-means clustering be generalized to operate solely on a distance matrix without requiring vector representations of data points?
• RQ2 How can the distance from a data point to its cluster centroid be computed when only a squared distance matrix is available?
• RQ3 What conditions must a distance matrix satisfy to ensure that the generalized centroid distances remain non-negative and interpretable?
• RQ4 Can a non-Euclidean distance matrix be corrected to support valid k-means clustering while preserving the original pairwise distances as much as possible?
• RQ5 How does the performance and interpretability of the generalized k-means method compare to standard k-means on real-world non-Euclidean data?

Key findings

• The paper successfully generalizes k-means to work with arbitrary distance matrices by expressing centroid distances as quadratic forms over the matrix.
• The method enables k-means clustering on data with no underlying Euclidean representation, such as protein sequences or other abstract data types.
• The generalized distance formula $ ||p_i - z_i||^2 = -\frac{1}{2} \lambda^T A \lambda $ is valid even when $ A $ is not derived from Euclidean space.
• Negative distances can arise if the quadratic form is not negative semi-definite on the zero-sum hyperplane, which invalidates the metric interpretation.
• A matrix correction method, such as $ \beta $-spread transformation, can be applied to eliminate negative distances while minimizing distortion of the original distance matrix.
• The approach is computationally more expensive than standard k-means due to repeated quadratic form evaluations, but it remains feasible for heuristic k-means algorithms.