[Paper Review] Performance Evaluation of Apache Spark MLlib Algorithms on an Intrusion Detection Dataset

Ramin Atefinia, Mahmood Ahmadi|arXiv (Cornell University)|Dec 10, 2022

Network Security and Intrusion Detection4 citations

TL;DR

This paper evaluates Apache Spark MLlib's performance on the CSE-CIC-IDS2018 dataset using six machine learning algorithms for intrusion detection. It demonstrates that Spark enables significant training speedup—especially for Naive Bayes and Random Forest—while maintaining 100% accuracy through distributed hyperparameter tuning, with optimal scalability observed up to 8 worker nodes.

ABSTRACT

The increase in the use of the Internet and web services and the advent of the fifth generation of cellular network technology (5G) along with ever-growing Internet of Things (IoT) data traffic will grow global internet usage. To ensure the security of future networks, machine learning-based intrusion detection and prevention systems (IDPS) must be implemented to detect new attacks, and big data parallel processing tools can be used to handle a huge collection of training data in these systems. In this paper Apache Spark, a general-purpose and fast cluster computing platform is used for processing and training a large volume of network traffic feature data. In this work, the most important features of the CSE-CIC-IDS2018 dataset are used for constructing machine learning models and then the most popular machine learning approaches, namely Logistic Regression, Support Vector Machine (SVM), three different Decision Tree Classifiers, and Naive Bayes algorithm are used to train the model using up to eight number of worker nodes. Our Spark cluster contains seven machines acting as worker nodes and one machine is configured as both a master and a worker. We use the CSE-CIC-IDS2018 dataset to evaluate the overall performance of these algorithms on Botnet attacks and distributed hyperparameter tuning is used to find the best single decision tree parameters. We have achieved up to 100% accuracy using selected features by the learning method in our experiments

Motivation & Objective

• To accelerate the training of machine learning models for intrusion detection using Apache Spark’s distributed computing framework.
• To address memory bottlenecks in resource-intensive algorithms like SVM and deep decision trees through cluster-based parallelization.
• To compare the scalability and performance of six MLlib algorithms—Logistic Regression, SVM, Naive Bayes, Single Decision Tree, Random Forest, and Gradient-Boosted Tree—on large-scale network traffic data.
• To replace manual hyperparameter tuning with a distributed, Spark-native hyperparameter tuning approach for improved model efficiency.
• To identify the most scalable and accurate algorithm for anomaly-based intrusion detection in big data environments.

Proposed method

• Utilized a Spark cluster with 7 worker nodes and 1 master node to distribute training of machine learning models on the CSE-CIC-IDS2018 dataset.
• Preprocessed the dataset by selecting key features using the 'learner' method to reduce dimensionality and improve model efficiency.
• Applied distributed hyperparameter tuning via Spark’s CrossValidator and TrainValidationSplit tools to optimize single decision tree parameters.
• Trained six MLlib algorithms: Logistic Regression, LinearSVC, Bernoulli and Multinomial Naive Bayes, Single Decision Tree, Random Forest, and Gradient-Boosted Tree.
• Measured training time, accuracy, precision, recall, and F1-score across varying numbers of worker nodes (1 to 8) to assess scalability and performance.
• Used Spark’s DataFrame API to streamline feature engineering, model training, and prediction pipeline integration.

Experimental results

Research questions

• RQ1 How does Apache Spark’s parallel processing capability improve training time for machine learning models on large-scale intrusion detection datasets?
• RQ2 Which MLlib algorithms exhibit the highest scalability and speedup when distributed across multiple worker nodes in a Spark cluster?
• RQ3 Can distributed hyperparameter tuning in Spark achieve better model accuracy and efficiency compared to manual tuning?
• RQ4 What is the impact of increasing worker nodes on model accuracy and training time for memory-intensive algorithms like SVM and deep decision trees?
• RQ5 Which algorithm achieves the highest accuracy (e.g., 100%) while maintaining strong scalability across different cluster configurations?

Key findings

• Naive Bayes achieved the highest speedup due to independent probability calculations, showing strong linear scalability across worker nodes.
• Training time for Logistic Regression decreased from 4653.09 seconds (single node) to 2436.68 seconds (three-node cluster), demonstrating effective parallelization.
• Support Vector Machine and Single Decision Tree showed diminishing returns beyond 3–4 worker nodes due to communication overhead and algorithmic constraints.
• Gradient-Boosted Tree achieved 99.6% precision, recall, and F1-score with 2045.21 seconds training time on 8 nodes, indicating high accuracy with moderate scalability.
• Random Forest showed higher variance in classification results due to random subspace selection, but maintained stable performance across cluster sizes.
• The best model achieved 100% accuracy using selected features and tuned hyperparameters, confirming the effectiveness of Spark-based hyperparameter tuning.