[论文解读] A Unified Subspace Outlier Ensemble Framework for Outlier Detection in High Dimensional Spaces

Zengyou He, Xiaofei Xu+1 more|ArXiv.org|May 24, 2005

Anomaly Detection Techniques and Applications被引用 3

一句话总结

本文提出了一种统一的子空间异常检测集成框架，用于高维异常检测，通过集成学习将异常得分建模为子空间特定因子的融合。SOE1算法仅使用一维子空间，即可在大规模分类数据集上实现当前最优的异常检测性能，同时速度比现有方法快一个数量级。

ABSTRACT

The task of outlier detection is to find small groups of data objects that are exceptional when compared with rest large amount of data. Detection of such outliers is important for many applications such as fraud detection and customer migration. Most such applications are high dimensional domains in which the data may contain hundreds of dimensions. However, the outlier detection problem itself is not well defined and none of the existing definitions are widely accepted, especially in high dimensional space. In this paper, our first contribution is to propose a unified framework for outlier detection in high dimensional spaces from an ensemble-learning viewpoint. In our new framework, the outlying-ness of each data object is measured by fusing outlier factors in different subspaces using a combination function. Accordingly, we show that all existing researches on outlier detection can be regarded as special cases in the unified framework with respect to the set of subspaces considered and the type of combination function used. In addition, to demonstrate the usefulness of the ensemble-learning based outlier detection framework, we developed a very simple and fast algorithm, namely SOE1 (Subspace Outlier Ensemble using 1-dimensional Subspaces) in which only subspaces with one dimension is used for mining outliers from large categorical datasets. The SOE1 algorithm needs only two scans over the dataset and hence is very appealing in real data mining applications. Experimental results on real datasets and large synthetic datasets show that: (1) SOE1 has comparable performance with respect to those state-of-art outlier detection algorithms on identifying true outliers and (2) SOE1 can be an order of magnitude faster than one of the fastest outlier detection algorithms known so far.

研究动机与目标

• 为高维空间中的异常检测缺乏广泛接受的定义提供解决方案。
• 在单一集成学习框架下统一现有的异常检测方法。
• 针对大规模分类数据集，开发一种基于最小子空间维度的快速、可扩展算法。
• 证明基于子空间的集成方法可在显著降低计算成本的同时实现高精度。

提出的方法

• 该框架将异常得分定义为在多个子空间中计算的异常因子的组合。
• 采用组合函数融合来自不同子空间的异常得分，从而实现对现有方法的统一处理。
• SOE1算法仅将子空间限制为一维，简化计算并减少I/O开销。
• SOE1仅对数据集进行两次完整扫描，使其在大规模数据挖掘中极为高效。
• 异常检测基于一维子空间中的基于频率的异常因子，通过组合函数聚合。
• 通过调整子空间集合和组合函数，该框架可将现有方法作为特例推广。

实验结果

研究问题

• RQ1 如何在统一框架下形式化高维空间中的异常检测？
• RQ2 在子空间上进行集成学习能否提升检测的准确性和效率？
• RQ3 仅使用一维子空间时，精度与速度之间的性能权衡如何？
• RQ4 所提出的框架与现有异常检测算法之间有何关系，并如何对其进行推广？

主要发现

• SOE1在真实和合成数据集上实现了与当前最优算法相当的异常检测性能。
• SOE1比现有最快异常检测算法之一快一个数量级。
• 统一框架可将所有现有异常检测方法作为特例，基于子空间选择和组合函数进行涵盖。
• SOE1中使用一维子空间可实现高效的两遍扫描算法，适用于大规模数据挖掘。