[Paper Review] Extending Isolation Forest for Anomaly Detection in Big Data via K-Means

Tahmid Rahman Laskar, Jimmy Xiangji Huang+6 more|arXiv (Cornell University)|Apr 27, 2021

Network Security and Intrusion Detection89 references4 citations

TL;DR

This paper proposes IForest-KMeans, a novel unsupervised anomaly detection framework that combines Isolation Forest with K-Means clustering to enable effective, real-time anomaly detection in large-scale industrial network traffic data. By leveraging Apache Spark for distributed processing and resolving training instability in Isolation Forest on big data, the method achieves comparable performance to state-of-the-art models while requiring fewer hyperparameters and enabling deployment on live streaming data.

ABSTRACT

Industrial Information Technology (IT) infrastructures are often vulnerable to cyberattacks. To ensure security to the computer systems in an industrial environment, it is required to build effective intrusion detection systems to monitor the cyber-physical systems (e.g., computer networks) in the industry for malicious activities. This paper aims to build such intrusion detection systems to protect the computer networks from cyberattacks. More specifically, we propose a novel unsupervised machine learning approach that combines the K-Means algorithm with the Isolation Forest for anomaly detection in industrial big data scenarios. Since our objective is to build the intrusion detection system for the big data scenario in the industrial domain, we utilize the Apache Spark framework to implement our proposed model which was trained in large network traffic data (about 123 million instances of network traffic) stored in Elasticsearch. Moreover, we evaluate our proposed model on the live streaming data and find that our proposed system can be used for real-time anomaly detection in the industrial setup. In addition, we address different challenges that we face while training our model on large datasets and explicitly describe how these issues were resolved. Based on our empirical evaluation in different use-cases for anomaly detection in real-world network traffic data, we observe that our proposed system is effective to detect anomalies in big data scenarios. Finally, we evaluate our proposed model on several academic datasets to compare with other models and find that it provides comparable performance with other state-of-the-art approaches.

Motivation & Objective

• Address the challenge of training Isolation Forest on large-scale, unlabeled industrial network traffic data where labeled data are unavailable.
• Overcome instability and scalability issues in Isolation Forest when applied to big data by integrating K-Means clustering for data partitioning and preprocessing.
• Enable real-time anomaly detection on live streaming network data in industrial cyber-physical systems.
• Demonstrate the effectiveness of the proposed method on both real-world large-scale datasets (123M instances) and academic benchmarks, achieving performance comparable to state-of-the-art models.
• Provide a replicable framework for anomaly detection in big data scenarios by resolving key training challenges in Isolation Forest.

Proposed method

• Apply K-Means clustering to partition large-scale network traffic data into smaller, more manageable clusters before feeding them into Isolation Forest.
• Use Apache Spark to distribute the training and inference pipeline, enabling scalable processing of 123 million network traffic instances.
• Integrate the K-Means centroids as additional features in the Isolation Forest model to improve anomaly detection by capturing local data structure.
• Optimize hyperparameters such as CR (contamination rate) and AQRE (anomaly quantile ratio estimate) to balance detection performance and model stability.
• Train the model on unlabeled data using the unsupervised nature of Isolation Forest, with K-Means reducing data complexity and improving convergence.
• Deploy the system for real-time inference on live streaming data, validating its performance in industrial deployment settings.

Figure 1: Normal network traffic pattern at iSecurity. Y-axis represents the total number of events occurred on each timestamp represented by the X-axis.

Experimental results

Research questions

• RQ1 Can combining K-Means clustering with Isolation Forest improve the scalability and stability of anomaly detection in large-scale industrial network traffic data?
• RQ2 How does the proposed IForest-KMeans framework perform on real-world big data compared to traditional Isolation Forest and other state-of-the-art models?
• RQ3 To what extent does the integration of K-Means reduce the dependency on critical hyperparameters like CR and AQRE in Isolation Forest?
• RQ4 Can the proposed system achieve real-time anomaly detection on live streaming network data in industrial environments?
• RQ5 What are the key challenges in training Isolation Forest on large datasets, and how can they be effectively mitigated using clustering-based preprocessing?

Key findings

• The proposed IForest-KMeans model achieved a 48% performance improvement in the Breastw dataset when CR was increased from 0.1 to 0.4, demonstrating sensitivity to hyperparameter tuning.
• The performance of standard Isolation Forest deteriorated significantly when AQRE was increased from 0.1 to 0.5, with a statistically significant p-value ≤ 0.05, highlighting its instability on large datasets.
• Despite using fewer hyperparameters, IForest-KMeans achieved performance comparable to Isolation Forest on 12 academic datasets, indicating robustness and generalization.
• The system successfully detected anomalies in real-time on live streaming data at iSecurity, proving its viability for industrial deployment.
• The model was trained on 123 million network traffic instances stored in Elasticsearch, confirming scalability and effectiveness in real-world big data scenarios.
• The paired t-test confirmed that performance differences between default and optimal hyperparameter settings were statistically significant (p ≤ 0.05), validating the importance of tuning.

Figure 2: Network traffic pattern during a cyberattack targeting the port 3389 using the TCP protocol on January 16th: the sudden spike in the number of events at 1 to 2 am and 5 to 6 am are flagged as cyberattacks by the network expert at iSecurity. At any other time interval, there are no such spi