[论文解读] Massive-scale Online Feature Selection for Sparse Ultra-high Dimensional Data.

Yue Wu, Steven C. H. Hoi+1 more|arXiv (Cornell University)|Sep 27, 2014

Face and Expression Recognition参考文献 24被引用 10

一句话总结

本文提出了一种二阶在线特征选择方法，通过优化计算实现超高维稀疏数据的高精度与极高效处理。该方法在单台机器上仅用数分钟即可处理十亿规模的特征数据集，其速度超越了批量方法与在线方法，同时保持了最先进水平的精度。

ABSTRACT

Abstract—Feature Selection is an important technique in machine learning and pattern classification, especially for handling high-dimensional data. Most existing studies have been restricted to batch learning, which is often inefficient and poorly scalable when handling big data in real world, especially when data arrives sequentially. Recent years have witnessed some emerging feature selection techniques using online learning. Despite enjoying significant advantages in efficiency and scalability, the existing online feature selection methods are not always accurate enough, and still not sufficiently fast when handling massive-scale data with ultra-high dimensionality. To address the limitations, we propose a novel online feature selection method by exploiting second-order information with optimized implementations, which not only improves the learning efficacy, but also significantly enhances computational efficiency. We conduct extensive experiments for evaluating both learning accuracy and time cost of different algorithms on massive-scale synthetic and real-world datasets, including a dataset with billion-scale features. Our results show that our technique achieves highly competitive accuracy as compared with state-of-the-art batch feature selection methods, but consumes significantly low computational cost that is orders of magnitude lower than both state-of-the-art batch and online feature selection methods. On a billion-scale synthetic dataset (1-billion dimensions, 1-billion nonzero features, and 1-million samples), our algorithm took only eight minutes with a normal single machine. Index Terms—Feature selection, online learning, sparsity, second-order online learning, ultra-high dimensionality F 1

研究动机与目标

• 解决现有批量与在线特征选择方法在大规模、超高维数据上存在的效率与可扩展性局限。
• 提升稀疏高维数据集在线特征选择中的学习精度与计算效率。
• 实现在高达十亿维特征空间数据上的实时、可扩展特征选择。
• 优化稀疏数据结构下的二阶在线学习技术，以降低计算成本而不损失精度。

提出的方法

• 该方法采用二阶在线学习，捕捉损失曲面中的曲率信息，相较于一阶方法具有更优的收敛性与精度。
• 集成针对稀疏数据结构定制的优化实现，以减少内存访问与计算开销。
• 算法按顺序处理特征，利用基于Hessian的近似方法增量更新特征权重，提升学习稳定性。
• 通过维护紧凑且稀疏的特征权重与梯度表示，最小化存储与计算开销。
• 该框架支持数据流上的增量学习，适用于现实世界中动态变化的数据环境。
• 该方法设计为可在单台机器上高效扩展，避免为超高维问题引入分布式系统。

实验结果

研究问题

• RQ1 相较于一阶在线方法，二阶在线学习是否能在超高维稀疏数据中提升特征选择的精度？
• RQ2 当处理高达十亿个特征的数据集时，该方法在时间和内存上的可扩展性如何？
• RQ3 经过优化的二阶在线学习在保持与批量方法相当精度的前提下，能否将计算成本降低数个数量级？
• RQ4 该方法在具有极端稀疏性与高维度的真实世界与合成数据集上的表现如何？

主要发现

• 在十亿规模的合成数据集上（10亿维，100万样本，10亿个非零特征），该算法仅用单台机器在八分钟内完成特征选择。
• 尽管为在线方法，该方法仍实现了与最先进批量特征选择方法相当的学习精度。
• 计算成本显著低于最先进的批量与在线特征选择方法，达到数个数量级的降低。
• 该方法在稀疏超高维数据上展现出强大的可扩展性与效率，时间效率优于现有方法。
• 优化实现显著减少了内存访问与计算时间，使大规模数据的实时处理成为可能。