[Paper Review] Strong Consistency of Reduced K-means Clustering

Yoshikazu Terada|arXiv (Cornell University)|Dec 20, 2012

Face and Expression Recognition13 references4 citations

TL;DR

This paper establishes the strong consistency of Reduced K-means (RKM) clustering under i.i.d. sampling by extending Pollard’s framework for k-means consistency. It proves that the RKM estimator almost surely converges to the population minimizers as sample size increases, and proposes a consistent criterion for selecting the optimal subspace dimension given the number of clusters.

ABSTRACT

Reduced k-means clustering is a method for clustering objects in a low-dimensional subspace. The advantage of this method is that both clustering of objects and low-dimensional subspace reflecting the cluster structure are simultaneously obtained. In this paper, the relationship between conventional k-means clustering and reduced k-means clustering is discussed. Conditions ensuring almost sure convergence of the estimator of reduced k-means clustering as unboundedly increasing sample size have been presented. The results for a more general model considering conventional k-means clustering and reduced k-means clustering are provided in this paper. Moreover, a new criterion and its consistent estimator are proposed to determine the optimal dimension number of a subspace, given the number of clusters.

Motivation & Objective

• To establish the theoretical foundation for Reduced K-means clustering by proving its strong consistency under i.i.d. sampling.
• To address the lack of statistical properties in RKM clustering, which is otherwise a descriptive method.
• To resolve the rotational indeterminacy issue in RKM by deriving conditions for the existence of population global minimizers.
• To propose a new consistent criterion and its estimator for determining the optimal number of dimensions in the low-dimensional subspace.
• To demonstrate through numerical experiments that the proposed criterion effectively identifies the optimal subspace dimension

Proposed method

• Uses the framework of the strong law of large numbers (SLLN) and uniform SLLN to analyze the convergence of the RKM objective function.
• Applies the Blum-DeHardt uniform SLLN, requiring only stationarity and ergodicity, to ensure almost sure convergence of the empirical estimator.
• Defines the RKM objective function as minimizing the average within-cluster sum of squares in a low-dimensional subspace: $ RKM_n = \frac{1}{n}\sum_{i=1}^n \min_{1\leq j\leq k} \|\mathbf{x}_i - A\mathbf{f}_j\|^2 $, where $ A $ is orthonormal and $ \mathbf{f}_j \in \mathbb{R}^q $.
• Introduces a new criterion $ \widehat{VR}(q) $ and its consistent estimator to select the optimal subspace dimension $ q $, given the number of clusters $ k $.
• Employs alternating least squares to solve the RKM optimization problem, iteratively updating cluster assignments, centroids, and subspace projection matrix.
• Validates the proposed criterion using adjusted Rand index (ARI) comparisons across different $ q $ values in simulated data with known cluster structures.

Experimental results

Research questions

• RQ1 Under what conditions does the Reduced K-means clustering estimator converge almost surely to the population minimizers as sample size increases?
• RQ2 How can the rotational indeterminacy of RKM be addressed in the theoretical consistency analysis?
• RQ3 What conditions ensure the existence of global minimizers in the population-level RKM objective function?
• RQ4 Can a consistent estimator be developed to select the optimal number of dimensions $ q $ in the subspace for a given number of clusters $ k $?
• RQ5 How effective is the proposed criterion in identifying the true subspace dimension compared to alternative methods like tandem clustering?

Key findings

• The RKM clustering estimator is strongly consistent under i.i.d. sampling, meaning it almost surely converges to the population minimizers as $ n \to \infty $, extending Pollard’s k-means consistency framework.
• The paper derives sufficient conditions for the existence of population global minimizers, overcoming the challenge of rotational indeterminacy in RKM.
• The proposed criterion $ \widehat{VR}(q) $ and its consistent estimator successfully identify the optimal subspace dimension, with agreement rates of 84% to 95% across 1000 simulated datasets.
• Numerical experiments show that RKM with optimal $ q $ achieves an adjusted Rand index (ARI) of 0.99, significantly outperforming tandem clustering (ARI = 0.26) when the first two principal components fail to reflect cluster structure.
• The rate of convergence for the RKM estimator is bounded under the assumption that the population distribution has bounded support, with explicit concentration bounds derived from Linder et al. (1994).
• The method is applicable to stationary ergodic processes, not limited to i.i.d. sampling, broadening its theoretical scope.