[Paper Review] Evaluation of Machine Learning Classifiers for Zero-Day Intrusion Detection -- An Analysis on CIC-AWS-2018 dataset

Qianru Zhou, Dimitrios P. Pezaros|arXiv (Cornell University)|May 9, 2019

Network Security and Intrusion Detection9 references52 citations

TL;DR

The paper evaluates six ML classifiers on the CIC-AWS-2018 dataset for zero-day intrusion detection using CICFlowMeter features, finding simple models like decision trees can achieve high accuracy.

ABSTRACT

Detecting Zero-Day intrusions has been the goal of Cybersecurity, especially intrusion detection for a long time. Machine learning is believed to be the promising methodology to solve that problem, numerous models have been proposed but a practical solution is still yet to come, mainly due to the limitation caused by the out-of-date open datasets available. In this paper, we take a deep inspection of the flow-based statistical data generated by CICFlowMeter, with six most popular machine learning classification models for Zero-Day attacks detection. The training dataset CIC-AWS-2018 Dataset contains fourteen types of intrusions, while the testing datasets contains eight different types of attacks. The six classification models are evaluated and cross validated on CIC-AWS-2018 Dataset for their accuracy in terms of false-positive rate, true-positive rate, and time overhead. Testing dataset, including eight novel (or Zero-Day) real-life attacks and benign traffic flows collected in real research production network are used to test the performance of the chosen decision tree classifier. Promising results are received with the accuracy as high as 100% and reasonable time overhead. We argue that with the statistical data collected from CICFlowMeter, simple machine learning models such as the decision tree classification could be able to take charge in detecting Zero-Day attacks.

Motivation & Objective

• Assess zero-day intrusion detection performance using flow-based statistical data from CICFlowMeter.
• Compare six machine learning classifiers on the CIC-AWS-2018 training/testing split.
• Analyze accuracy, false-positive/true-positive rates, and time overhead for each model.
• Identify whether simple models can effectively detect zero-day attacks.

Proposed method

• Use CICFlowMeter to extract flow-based statistical features from network data.
• Evaluate six ML classification models on the CIC-AWS-2018 dataset with 14 training intrusion types and 8 testing types.
• Perform cross-validation to measure accuracy, false-positive rate, true-positive rate, and time overhead.
• Test the chosen model on real-life zero-day attacks and benign traffic to assess practical performance.
• Compare results to determine if simple models (e.g., decision tree) are viable for zero-day detection.

Experimental results

Research questions

• RQ1 Which ML classifiers provide the best balance of accuracy and overhead for zero-day intrusion detection on CIC-AWS-2018?
• RQ2 Can simple models like decision trees achieve competitive performance against more complex models on this dataset?
• RQ3 How do the models perform against zero-day attacks and benign traffic in real production network data?

Key findings

• Six ML models were evaluated on the CIC-AWS-2018 dataset.
• Training set contains fourteen intrusion types; testing set contains eight intrusion types.
• Testing with real zero-day attacks and benign traffic showed promising results with accuracy up to 100%.
• Time overhead of the models remains reasonable, supporting practical deployment.
• The study argues that simple models such as decision trees can effectively detect zero-day attacks using CICFlowMeter features.