[Paper Review] Strong Consistency of Factorial K-means Clustering

Yoshikazu Terada|arXiv (Cornell University)|Jan 4, 2013

Advanced Clustering Algorithms Research9 references4 citations

TL;DR

This paper establishes the strong consistency of Factorial K-means (FKM) clustering under i.i.d. sampling by proving that the sample estimator almost surely converges to the population minimizer as sample size increases. The proof extends frameworks from k-means and reduced k-means clustering, leveraging uniform strong law of large numbers and continuity of the objective function on compact parameter sets.

ABSTRACT

Factorial k-means (FKM) clustering is a method for clustering objects in a low-dimensional subspace. The advantage of this method is that the partition of objects and the low-dimensional subspace reflecting the cluster structure are obtained, simultaneously. Conditions that ensure the almost sure convergence of the estimator of FKM clustering as the sample size increases unboundedly are derived. The result is proved for a more general model including FKM clustering.

Motivation & Objective

• To establish the strong consistency of Factorial K-means (FKM) clustering under i.i.d. sampling conditions.
• To derive sufficient conditions ensuring the existence of population-level global minimizers for the FKM objective function.
• To extend the theoretical consistency framework used in k-means and reduced k-means clustering to the FKM setting.
• To prove uniform almost sure convergence of the FKM objective function and continuity of its minimizers.
• To provide a foundation for future work on the rate of convergence of FKM estimators.

Proposed method

• Formalizes FKM clustering as a minimization of the empirical loss function: $ FKM_n(F, A) = \frac{1}{n} \sum_{i=1}^n \min_{1 \leq j \leq k} \|A^T \mathbf{x}_i - \mathbf{f}_j\|^2 $, where $ A $ is a $ p \times q $ orthonormal matrix and $ \mathbf{f}_j \in \mathbb{R}^q $.
• Uses the strong law of large numbers (SLLN) to show almost sure convergence of the empirical objective function to its population counterpart: $ \lim_{n \to \infty} FKM(F, A, P_n) = FKM(F, A, P) \text{ a.s.} $
• Establishes that the optimal cluster centers $ F_n $ eventually lie within a compact set $ \mathcal{R}_k^*(5M) $, ensuring tightness of the estimator.
• Applies the Blum-DeHardt uniform SLLN to prove uniform convergence of the objective function over compact parameter spaces.
• Uses continuity of the population objective function $ \Psi(\cdot, P) $ on compact sets to establish convergence of minimizers.
• Implements a perturbation argument via $ \tilde{\theta}_n $, which matches the estimator $ \hat{\theta}_n $ for large $ n $, to prove $ \lim_{n \to \infty} d(\hat{\theta}_n, \Theta') = 0 \text{ a.s.} $

Experimental results

Research questions

• RQ1 Under what conditions does the FKM clustering estimator converge almost surely to the population minimizer?
• RQ2 What conditions ensure the existence of a global minimizer for the population FKM objective function?
• RQ3 How does the strong consistency of FKM clustering compare to that of k-means and reduced k-means clustering?
• RQ4 Can the uniform SLLN and continuity of the FKM objective function be established over compact parameter sets?
• RQ5 What is the theoretical foundation for the convergence of FKM estimators under minimal probabilistic assumptions?

Key findings

• The sample FKM estimator converges almost surely to the population minimizer as the sample size increases, establishing strong consistency.
• The optimal cluster centers $ F_n $ are almost surely contained within a compact set $ \mathcal{R}_k^*(5M) $ for sufficiently large $ n $, ensuring estimator tightness.
• The population objective function $ \Psi(\cdot, P) $ is continuous on the compact parameter space $ \Theta_k^*(5M) $, enabling convergence of minimizers.
• The uniform SLLN holds for the FKM objective function over compact sets, ensuring uniform convergence of empirical to population loss.
• The distance between the sample estimator $ \hat{\theta}_n $ and the set of population minimizers $ \Theta' $ converges to zero almost surely.
• The result holds under i.i.d. sampling, and the proof framework extends to stationary and ergodic processes, not requiring i.i.d. assumptions beyond what is needed for SLLN.