[Paper Review] Evaluating Standard Feature Sets Towards Increased Generalisability and Explainability of ML-based Network Intrusion Detection

Mohanad Sarhan, Siamak Layeghy+1 more|arXiv (Cornell University)|Apr 15, 2021

Network Security and Intrusion Detection4 citations

TL;DR

This paper evaluates the generalisability and explainability of ML-based network intrusion detection by comparing NetFlow and CICFlowMeter feature sets across three benchmark datasets (CSE-CIC-IDS2018, ToN-IoT, BoT-IoT). Using Random Forest and Deep Feed-Forward classifiers, it demonstrates that NetFlow features consistently outperform CICFlowMeter in detection accuracy and inference speed, while SHAP analysis reveals that forward-directional and IAT-based features are most influential in attack detection across datasets.

ABSTRACT

Machine Learning (ML)-based network intrusion detection systems bring many benefits for enhancing the cybersecurity posture of an organisation. Many systems have been designed and developed in the research community, often achieving a close to perfect detection rate when evaluated using synthetic datasets. However, the high number of academic research has not often translated into practical deployments. There are several causes contributing towards the wide gap between research and production, such as the limited ability of comprehensive evaluation of ML models and lack of understanding of internal ML operations. This paper tightens the gap by evaluating the generalisability of a common feature set to different network environments and attack scenarios. Therefore, two feature sets (NetFlow and CICFlowMeter) have been evaluated in terms of detection accuracy across three key datasets, i.e., CSE-CIC-IDS2018, BoT-IoT, and ToN-IoT. The results show the superiority of the NetFlow feature set in enhancing the ML models detection accuracy of various network attacks. In addition, due to the complexity of the learning models, SHapley Additive exPlanations (SHAP), an explainable AI methodology, has been adopted to explain and interpret the classification decisions of ML models. The Shapley values of two common feature sets have been analysed across multiple datasets to determine the influence contributed by each feature towards the final ML prediction.

Motivation & Objective

• To address the gap between academic research and real-world deployment of ML-based network intrusion detection systems (NIDS).
• To evaluate the generalisability of standard feature sets—NetFlow and CICFlowMeter—across diverse network environments and attack scenarios.
• To improve model explainability by applying SHapley Additive exPlanations (SHAP) to interpret feature contributions in ML predictions.
• To generate and release two new CICFlowMeter-formatted datasets (CIC-BoT-IoT and CIC-ToN-IoT) for standardized, cross-dataset evaluation.

Proposed method

• Constructed two new datasets—CIC-BoT-IoT and CIC-ToN-IoT—by converting ToN-IoT and BoT-IoT into CICFlowMeter feature format for consistent evaluation.
• Applied two machine learning models—Random Forest (RF) and Deep Feed-Forward (DFF)—to classify network traffic across all three datasets using both NetFlow and CICFlowMeter feature sets.
• Used SHapley Additive exPlanations (SHAP) to compute feature importance, with KernelSHAP for DFF and TreeSHAP for RF, to interpret model decisions.
• Normalised mean SHAP values across test samples and visualised feature influence rankings to compare impact across datasets and models.
• Evaluated detection accuracy and inference time to assess model performance and efficiency under different feature sets.
• Released all datasets and feature sets publicly to enable reproducible, cross-dataset benchmarking of ML-based NIDS.

Figure 1 : Dataset Generation Process

Experimental results

Research questions

• RQ1 Does the NetFlow feature set generalise better than the CICFlowMeter feature set across diverse network environments and attack types?
• RQ2 How do detection accuracy and inference time compare between NetFlow and CICFlowMeter when used with different ML models (RF and DFF)?
• RQ3 Which network features contribute most significantly to ML model predictions across multiple datasets, as revealed by SHAP analysis?
• RQ4 To what extent do feature importance rankings vary across datasets, and are there consistent high-impact features across different attack scenarios?

Key findings

• The NetFlow feature set achieved consistently higher detection accuracy than CICFlowMeter across all three datasets (CSE-CIC-IDS2018, ToN-IoT, BoT-IoT) with both RF and DFF classifiers.
• NetFlow-based models demonstrated lower prediction times, indicating better efficiency in real-time deployment scenarios.
• Forward-directional features (e.g., 'Fwd IAT Min', 'Fwd IAT Mean') were among the top five most influential features in the CICFlowMeter-based CIC-BoT-IoT dataset for both RF and DFF models.
• The RF classifier was most influenced by forward-directional and IAT-based features in the CICFlowMeter format, while NetFlow features showed stronger consistency across models.
• NetFlow-based features had five common top-10 influential features across both RF and DFF models, compared to only two for CICFlowMeter, indicating greater stability and generalisability.
• SHAP analysis revealed significant variation in feature importance across datasets, suggesting that feature impact is highly dependent on the specific attack types and benign traffic patterns present.

(a) RF