[Paper Review] Scalable Kernel Clustering: Approximate Kernel k-means

Radha Chitta, Rong Jin+2 more|arXiv (Cornell University)|Feb 16, 2014

Face and Expression Recognition47 references18 citations

TL;DR

This paper proposes Approximate Kernel k-means (aKKm), a scalable kernel clustering algorithm that uses random sampling to approximate the kernel matrix and restricts cluster centers to a low-dimensional subspace spanned by a small set of sampled points. The method achieves clustering performance comparable to full kernel k-means with significantly reduced time and memory costs, and further improves accuracy using ensemble clustering techniques.

ABSTRACT

Kernel-based clustering algorithms have the ability to capture the non-linear structure in real world data. Among various kernel-based clustering algorithms, kernel k-means has gained popularity due to its simple iterative nature and ease of implementation. However, its run-time complexity and memory footprint increase quadratically in terms of the size of the data set, and hence, large data sets cannot be clustered efficiently. In this paper, we propose an approximation scheme based on randomization, called the Approximate Kernel k-means. We approximate the cluster centers using the kernel similarity between a few sampled points and all the points in the data set. We show that the proposed method achieves better clustering performance than the traditional low rank kernel approximation based clustering schemes. We also demonstrate that its running time and memory requirements are significantly lower than those of kernel k-means, with only a small reduction in the clustering quality on several public domain large data sets. We then employ ensemble clustering techniques to further enhance the performance of our algorithm.

Motivation & Objective

• To address the quadratic time and memory complexity of kernel k-means on large datasets by enabling scalable kernel-based clustering.
• To develop an efficient approximation scheme that avoids computing the full n×n kernel matrix while preserving clustering quality.
• To improve performance over existing low-rank kernel approximation methods through a novel use of eigenvectors in the Nystrom framework.
• To enhance clustering accuracy by integrating ensemble clustering techniques with the approximate kernel method.
• To provide theoretical bounds on kernel approximation error that are tighter than those of the naive Nystrom method.

Proposed method

• The method samples m ≪ n data points randomly and uses their kernel similarities to all points to form an n×m kernel matrix, reducing storage and computation.
• Cluster centers are approximated within the subspace spanned by the m sampled points, avoiding full kernel matrix computation.
• The algorithm leverages all eigenvectors of the approximate kernel matrix without explicitly computing them, improving accuracy over spectral clustering variants.
• It employs a tighter theoretical bound on kernel approximation error by analyzing the coherence of the kernel matrix and sampling distribution.
• Ensemble clustering is applied by running multiple instances with different random samples and combining results via meta-clustering to boost robustness and performance.
• The method is grounded in the Nystrom approximation framework, with theoretical guarantees derived from matrix concentration and spectral approximation theory.

Experimental results

Research questions

• RQ1 Can a kernel k-means variant be designed to scale efficiently to large datasets without computing the full kernel matrix?
• RQ2 How does the proposed approximation scheme compare in clustering accuracy to traditional low-rank kernel approximation methods?
• RQ3 Can theoretical bounds on kernel approximation error be improved over the naive Nystrom method?
• RQ4 Does integrating ensemble clustering techniques further enhance the performance of the approximate kernel k-means algorithm?
• RQ5 What is the trade-off between computational efficiency and clustering quality in the proposed method compared to kernel k-means and other scalable alternatives?

Key findings

• The proposed aKKm algorithm achieves clustering performance comparable to full kernel k-means on multiple public large-scale datasets, with only a small drop in accuracy.
• The method reduces computational complexity and memory usage from O(n²) to O(nm), where m ≪ n, resulting in significant speedups on large datasets.
• Theoretical analysis shows that the kernel approximation error bound is tighter than that of the naive Nystrom method, especially when the kernel matrix has low coherence.
• Empirical results demonstrate that aKKm outperforms other scalable kernel clustering methods, including Nystrom-based spectral clustering, in terms of clustering accuracy.
• The integration of ensemble clustering techniques further improves the robustness and performance of aKKm, particularly on noisy or complex data structures.
• The algorithm scales effectively to datasets with tens of thousands of points, maintaining high efficiency while preserving non-linear cluster structure.