[论文解读] F-tree: an algorithm for clustering transactional data using frequency tree

Mahmoud Mahdi, Samir Abdelrahman+2 more|arXiv (Cornell University)|May 1, 2017

Data Mining Algorithms and Applications被引用 7

一句话总结

F-Tree 是一种新颖的聚类算法，适用于高维、大规模的事务型数据。它借鉴 FP-Growth 的思想，利用频率树结构，优先构建小型、纯净的聚类，然后高效地进行合并。通过引入一种基于概率的重叠准则，F-Tree 在聚类速度和结果质量方面均优于直接聚类或迭代优化方法，显著缩短了运行时间并提升了聚类纯度。

ABSTRACT

Clustering is an important data mining technique that groups similar data records, recently categorical transaction clustering is received more attention. In this research, we study the problem of categorical data clustering for transactional data characterized with high dimensionality and large volume. We propose a novel algorithm for clustering transactional data called F-Tree, which is based on the idea of the frequent pattern algorithm FP-tree; the fastest approaches to the frequent item set mining. And the simple idea behind the F-Tree is to generate small high pure clusters, and then merge them. That makes it fast, and dynamic in clustering large transactional datasets with high dimensions. We also present a new solution to solve the overlapping problem between clusters, by defining a new criterion function, which is based on the probability of overlapping between weighted items. Our experimental evaluation on real datasets shows that: Firstly, F-Tree is effective in finding interesting clusters. Secondly, the usage of the tree structure reduces the clustering process time of the large data set with high attributes. Thirdly, the proposed evaluation metric used efficiently to solve the overlapping of transaction items generates high-quality clustering results. Finally, we have concluded that the process of merging pure and small clusters increases the purity of resulted clusters as well as it reduces the time of clustering better than the process of generating clusters directly from dataset then refine clusters.

研究动机与目标

• 为高效处理大规模、高维的分类事务型数据集聚类挑战提供解决方案。
• 通过基于树的频繁模式挖掘方法，减少高维数据的聚类时间并提升聚类质量。
• 通过基于加权项目概率的新准则，解决聚类中的重叠问题。
• 证明：先生成小型、纯净的聚类，再进行合并，其聚类纯度与性能优于直接聚类或迭代优化方法。

提出的方法

• F-Tree 从事务型数据构建频率树（F-Tree），类似于 FP-Growth 算法，以高效挖掘频繁项集。
• 基于 F-Tree 结构中的频繁模式，生成小型、高纯度的聚类，以降低计算开销。
• 采用一种新颖的合并策略，根据共享的频繁项将这些纯净聚类合并为最终聚类。
• 引入一种新的准则函数，用于计算加权项目之间重叠的概率，以解决聚类重叠问题。
• 算法采用基于频率的剪枝机制，避免冗余聚类的生成，提升可扩展性。
• 最终的聚类过程是动态且增量式的，适用于大规模事务型数据。

实验结果

研究问题

• RQ1 与传统方法相比，基于频率树的方法是否能实现对高维事务型数据更快的聚类？
• RQ2 先生成小型、纯净聚类并进行合并，是否能获得比直接聚类或迭代优化更高的聚类纯度？
• RQ3 基于概率的重叠准则是否能有效解决聚类间项目分配冲突？
• RQ4 F-Tree 结构在大规模、高维数据集上能在多大程度上减少聚类时间？

主要发现

• F-Tree 能够在真实事务型数据集中有效识别出有意义且有趣的聚类。
• 使用 F-Tree 结构显著减少了高维属性大规模数据集的聚类时间。
• 所提出的概率重叠准则能高效解决聚类重叠问题，从而获得更高质量的聚类结果。
• 与直接聚类或迭代优化相比，通过合并小型、纯净聚类可同时提升聚类纯度与执行效率。