[论文解读] Extending Isolation Forest for Anomaly Detection in Big Data via K-Means

Tahmid Rahman Laskar, Jimmy Xiangji Huang+6 more|arXiv (Cornell University)|Apr 27, 2021

Network Security and Intrusion Detection参考文献 89被引用 4

一句话总结

本文提出了一种名为IForest-KMeans的新颖无监督异常检测框架，通过将孤立森林（Isolation Forest）与K-Means聚类相结合，实现了在大规模工业网络流量数据中高效、实时的异常检测。该方法利用Apache Spark进行分布式处理，并解决了孤立森林在大规模数据上训练时的不稳定性问题，实现了与最先进模型相当的性能，同时所需超参数更少，且可部署于实时流式数据环境。

ABSTRACT

Industrial Information Technology (IT) infrastructures are often vulnerable to cyberattacks. To ensure security to the computer systems in an industrial environment, it is required to build effective intrusion detection systems to monitor the cyber-physical systems (e.g., computer networks) in the industry for malicious activities. This paper aims to build such intrusion detection systems to protect the computer networks from cyberattacks. More specifically, we propose a novel unsupervised machine learning approach that combines the K-Means algorithm with the Isolation Forest for anomaly detection in industrial big data scenarios. Since our objective is to build the intrusion detection system for the big data scenario in the industrial domain, we utilize the Apache Spark framework to implement our proposed model which was trained in large network traffic data (about 123 million instances of network traffic) stored in Elasticsearch. Moreover, we evaluate our proposed model on the live streaming data and find that our proposed system can be used for real-time anomaly detection in the industrial setup. In addition, we address different challenges that we face while training our model on large datasets and explicitly describe how these issues were resolved. Based on our empirical evaluation in different use-cases for anomaly detection in real-world network traffic data, we observe that our proposed system is effective to detect anomalies in big data scenarios. Finally, we evaluate our proposed model on several academic datasets to compare with other models and find that it provides comparable performance with other state-of-the-art approaches.

研究动机与目标

• 解决在缺乏标注数据的情况下，对大规模、无标签工业网络流量数据进行孤立森林训练的挑战。
• 通过引入K-Means聚类进行数据划分与预处理，克服孤立森林在应用于大数据时的不稳定性与可扩展性问题。
• 实现在工业信息物理系统中对实时流式网络数据的异常检测。
• 在真实世界大规模数据集（12300万条实例）和学术基准上验证所提方法的有效性，实现与最先进模型相当的性能。
• 通过解决孤立森林训练中的关键挑战，提供一种可在大数据场景中复现的异常检测框架。

提出的方法

• 在将数据输入孤立森林前，使用K-Means聚类将大规模网络流量数据划分为更小、更易管理的簇。
• 利用Apache Spark分发训练与推理流程，实现对1.23亿条网络流量实例的可扩展处理。
• 将K-Means的质心作为额外特征集成到孤立森林模型中，以捕捉局部数据结构，提升异常检测性能。
• 优化超参数（如CR（异常率）和AQRE（异常分位数估计比）），以平衡检测性能与模型稳定性。
• 利用孤立森林的无监督特性，在无标签数据上训练模型，同时K-Means降低数据复杂度并提升收敛性。
• 在实时流式数据上部署系统以实现在线推理，验证其在工业部署环境中的性能。

Figure 1: Normal network traffic pattern at iSecurity. Y-axis represents the total number of events occurred on each timestamp represented by the X-axis.

实验结果

研究问题

• RQ1 将K-Means聚类与孤立森林结合，能否提升大规模工业网络流量数据中异常检测的可扩展性与稳定性？
• RQ2 与传统孤立森林及其他最先进模型相比，所提出的IForest-KMeans框架在真实世界大数据上的表现如何？
• RQ3 K-Means的集成在多大程度上降低了孤立森林对关键超参数（如CR与AQRE）的依赖性？
• RQ4 所提系统能否在工业环境中实现对实时流式网络数据的异常检测？
• RQ5 在大规模数据集上训练孤立森林时面临哪些关键挑战，如何通过基于聚类的预处理有效缓解？

主要发现

• 当CR从0.1提升至0.4时，IForest-KMeans在Breastw数据集上实现了48%的性能提升，表明其对超参数调优具有显著敏感性。
• 当AQRE从0.1增至0.5时，标准孤立森林的性能显著下降，且p值≤0.05，具有统计显著性，凸显其在大规模数据上的不稳定性。
• 尽管使用更少的超参数，IForest-KMeans在12个学术数据集上的性能与孤立森林相当，表明其具备鲁棒性与泛化能力。
• 系统在iSecurity平台成功实现实时流式数据异常检测，证明其在工业部署中的可行性。
• 模型在Elasticsearch中存储的1.23亿条网络流量实例上完成训练，证实其在真实世界大数据场景下的可扩展性与有效性。
• 配对t检验确认，缺省与最优超参数设置之间的性能差异具有统计显著性（p ≤ 0.05），验证了调优的重要性。

Figure 2: Network traffic pattern during a cyberattack targeting the port 3389 using the TCP protocol on January 16th: the sudden spike in the number of events at 1 to 2 am and 5 to 6 am are flagged as cyberattacks by the network expert at iSecurity. At any other time interval, there are no such spi