[论文解读] Topics in matrix sampling algorithms

Petros Drineas, Christos Boutsidis|arXiv (Cornell University)|Jan 1, 2011

Sparse and Compressive Sensing Techniques参考文献 135被引用 12

一句话总结

本文提出了针对机器学习中三个核心问题的新型矩阵采样算法：低秩列基矩阵近似、最小二乘回归中的核压缩集构建，以及k均值聚类中的特征选择。通过选取少量列、行或特征，所提出的方法在近似质量上可与完整数据解相媲美，同时具备更优的理论保证，并首次将该方法拓展至k均值聚类的新应用领域。

ABSTRACT

We study three fundamental problems of Linear Algebra, lying in the heart of various Machine Learning applications, namely: (i) Low-rank Column-based Matrix Approximation, (ii) Coreset Construction in Least-Squares Regression, and (iii) Feature Selection in k-means Clustering. A high level description of these problems is as follows: given a matrix A and an integer r, what are the r most “important” columns (or rows) in A? A more detailed description is given momentarily. 1. Low-rank Column-based Matrix Approximation. We are given a matrix A and a target rank k. The goal is to select a subset of columns of A and, by using only these columns, compute a rank k approximation to A that is as good as the rank k approximation that would have been obtained by using all the columns. 2. Coreset Construction in Least-Squares Regression. We are given a matrix A and a vector b. Consider the (over-constrained) least-squares problem of minimizing ||Ax − b||2, over all vectors x ∈ D . The domain D represents the constraints on the solution and can be arbitrary. The goal is to select a subset of the rows of A and b and, by using only these rows, find a solution vector that is as good as the solution vector that would have been obtained by using all the rows. 3. Feature Selection in K-means Clustering. We are given a set of points described with respect to a large number of features. The goal is to select a subset of the features and, by using only this subset, obtain a k-partition of the points that is as good as the partition that would have been obtained by using all the features. We present novel algorithms for all three problems mentioned above. Our results can be viewed as follow-up research to a line of work known as “Matrix Sampling Algorithms”. Frieze et al. [59] presented the first such algorithm for the Low-rank Matrix Approximation problem. Since then, such algorithms have been developed for several other problems, e.g. Regression [47], Graph Sparsification [131], and Linear Equation Solving [128]. Our contributions to this line of research are: (i) improved algorithms for Low-rank Matrix Approximation and Regression (ii) algorithms for a new problem domain ( K-means Clustering).

研究动机与目标

• 开发改进的低秩列基矩阵近似与最小二乘回归核压缩集构建算法。
• 将矩阵采样技术拓展至新问题领域：k均值聚类中的特征选择。
• 在仅使用少量列、行或特征的前提下，提供近似质量的理论保证。
• 将矩阵采样框架从传统问题推广至包含聚类应用的更广范围。
• 提升大规模线性代数问题中子集选择的效率与准确性，这些问题在机器学习中具有核心地位。

提出的方法

• 利用矩阵采样技术，从矩阵A中选取少量具有代表性的列，用于低秩近似。
• 通过行与b中条目的采样策略，构建保持完整最小二乘问题解质量的核压缩集。
• 通过选取最小数量的特征来保留聚类结构，将基于采样的特征选择方法适配至k均值聚类。
• 采用基于矩阵结构导出的重要性评分的随机采样方法，确保近似精度。
• 通过理论分析，建立仅使用r个选定分量时，近似误差相对于最优解的边界。
• 将原始用于低秩近似的矩阵采样工作，通过统一的采样框架推广至回归与聚类问题。

实验结果

研究问题

• RQ1 矩阵采样技术能否改进，以在低秩列基矩阵近似中提供更优的近似保证？
• RQ2 基于采样的核压缩集构建能否拓展至最小二乘回归中的任意约束域？
• RQ3 矩阵采样能否被适配至k均值聚类中的特征选择问题，并提供理论性能保证？
• RQ4 所选列、行或特征的子集在近似质量上与完整解相比如何？
• RQ5 能否为所有三个问题中的采样近似误差建立理论边界？

主要发现

• 所提算法在低秩矩阵近似问题中，实现了与先前方法相当或更优的近似误差边界。
• 在最小二乘回归中，核压缩集构建仅使用原始行的极小部分，仍能保持解的质量。
• 该方法首次为k均值聚类中的基于采样的特征选择提供了理论框架。
• 理论分析证实，所选特征子集产生的聚类结果与使用全部特征所得结果非常接近。
• 该框架统一了多种问题中的采样策略，展示了超越原始低秩设置的广泛适用性。
• 研究结果将矩阵采样算法的应用范围拓展至聚类，开辟了重要的新应用领域。