[论文解读] Sparse K-Means with $\ell_{\infty}/\ell_0$ Penalty for High-Dimensional Data Clustering

Xiangyu Chang, Yu Wang+2 more|arXiv (Cornell University)|Mar 31, 2014

Advanced Clustering Algorithms Research参考文献 21被引用 5

一句话总结

本文提出了一种新颖的稀疏k-means框架，采用ℓ₀/ℓ∞惩罚以提升高维聚类中的特征选择性能。通过设计一种高效的迭代算法来求解ℓ₀-k-means，该方法实现了理论上的特征选择一致性，并在合成数据和生物数据上展现出优于ℓ₁-k-means的噪声特征检测能力。

ABSTRACT

Sparse clustering, which aims to find a proper partition of an extremely high-dimensional data set with redundant noise features, has been attracted more and more interests in recent years. The existing studies commonly solve the problem in a framework of maximizing the weighted feature contributions subject to a $\ell_2/\ell_1$ penalty. Nevertheless, this framework has two serious drawbacks: One is that the solution of the framework unavoidably involves a considerable portion of redundant noise features in many situations, and the other is that the framework neither offers intuitive explanations on why this framework can select relevant features nor leads to any theoretical guarantee for feature selection consistency. In this article, we attempt to overcome those drawbacks through developing a new sparse clustering framework which uses a $\ell_{\infty}/\ell_0$ penalty. First, we introduce new concepts on optimal partitions and noise features for the high-dimensional data clustering problems, based on which the previously known framework can be intuitively explained in principle. Then, we apply the suggested $\ell_{\infty}/\ell_0$ framework to formulate a new sparse k-means model with the $\ell_{\infty}/\ell_0$ penalty ($\ell_0$-k-means for short). We propose an efficient iterative algorithm for solving the $\ell_0$-k-means. To deeply understand the behavior of $\ell_0$-k-means, we prove that the solution yielded by the $\ell_0$-k-means algorithm has feature selection consistency whenever the data matrix is generated from a high-dimensional Gaussian mixture model. Finally, we provide experiments with both synthetic data and the Allen Developing Mouse Brain Atlas data to support that the proposed $\ell_0$-k-means exhibits better noise feature detection capacity over the previously known sparse k-means with the $\ell_2/\ell_1$ penalty ($\ell_1$-k-means for short).

研究动机与目标

• 解决现有稀疏k-means方法在保留大量冗余噪声特征方面的局限性。
• 克服ℓ₁-k-means中使用的ℓ₂/ℓ₁惩罚框架在理论和可解释性方面的不足。
• 基于ℓ₀/ℓ∞惩罚，开发一种新的稀疏聚类框架，以实现一致且直观的特征选择。
• 在高维高斯混合模型下，建立特征选择一致性的理论保证。
• 通过实证结果证明，所提出的ℓ₀-k-means在识别和剔除噪声特征方面优于ℓ₁-k-means。

提出的方法

• 为高维聚类问题引入最优划分和噪声特征的新定义。
• 提出一种新的稀疏k-means模型——ℓ₀-k-means，将其表述为带有ℓ₀/ℓ∞惩罚的优化问题，以促进稀疏性并选择相关特征。
• 设计一种高效的迭代算法，用于求解非凸的ℓ₀-k-means问题，从而实现实际应用。
• 通过理论分析证明，在高维高斯混合模型下，ℓ₀-k-means的解可实现特征选择一致性。
• 利用Stirling近似和集中不等式，界定高维设置下错误特征选择的概率。
• 对数据矩阵进行标准化，以简化分析，并确保理论结果在现实条件下成立。

实验结果

研究问题

• RQ1 基于ℓ₀/ℓ∞惩罚的新稀疏聚类框架能否克服ℓ₂/ℓ₁惩罚框架在保留噪声特征方面的局限性？
• RQ2 当数据由高维高斯混合模型生成时，ℓ₀-k-means方法是否能实现理论上的特征选择一致性？
• RQ3 在合成数据和真实世界数据上，ℓ₀-k-means在检测和剔除噪声特征方面的性能与ℓ₁-k-means相比如何？
• RQ4 随着样本量增加，ℓ₀-k-means框架下正确特征选择的理论概率是多少？
• RQ5 尽管ℓ₀惩罚具有非凸性，ℓ₀-k-means算法是否仍能高效实现并具有可解释性？

主要发现

• ℓ₀-k-means算法实现了特征选择一致性，即其渐近地仅选择相关特征，并以高概率排除所有噪声特征。
• 在高维高斯混合模型下，随着样本量增加，ℓ₀-k-means正确识别相关特征的概率趋近于1。
• 理论边界表明，若特征数量的增长速度慢于exp(n∑πₖμₖ²/258)，则正确特征选择的概率收敛于1。
• 在合成数据上的实证结果表明，ℓ₀-k-means能成功剔除ℓ₁-k-means无法剔除的噪声特征。
• 在Allen发育小鼠大脑图谱数据集上，ℓ₀-k-means展现出优于ℓ₁-k-means的噪声特征检测能力。
• 所提出的方法计算高效且可解释，为现有稀疏k-means方法提供了一种实用的替代方案。