[Paper Review] IoTGeM: Generalizable Models for Behaviour-Based IoT Attack Detection

Kahraman Kostas, Mike Just+1 more|ArXiv.org|Oct 17, 2023

Network Security and Intrusion Detection4 citations

TL;DR

IoTGeM proposes a generalizable, robust framework for behavior-based IoT attack detection using improved feature engineering and a multi-step feature selection process to reduce overfitting. By employing isolated training/testing splits, diverse ML models, and explainable AI (SHAP), the method achieves high detection accuracy across multiple datasets while ensuring reproducibility and model interpretability.

ABSTRACT

Previous research on behavior-based attack detection for networks of IoT devices has resulted in machine learning models whose ability to adapt to unseen data is limited and often not demonstrated. This paper presents IoTGeM, an approach for modeling IoT network attacks that focuses on generalizability, yet also leads to better detection and performance. We first introduce an improved rolling window approach for feature extraction. To reduce overfitting, we then apply a multi-step feature selection process where a Genetic Algorithm (GA) is uniquely guided by exogenous feedback from a separate, independent dataset. To prevent common data leaks that have limited previous models, we build and test our models using strictly isolated train and test datasets. The resulting models are rigorously evaluated using a diverse portfolio of machine learning algorithms and datasets. Our window-based models demonstrate superior generalization compared to traditional flow-based models, particularly when tested on unseen datasets. On these stringent, cross-dataset tests, IoTGeM achieves F1 scores of 99\% for ACK, HTTP, SYN, MHD, and PS attacks, as well as a 94\% F1 score for UDP attacks. Finally, we build confidence in the models by using the SHAP (SHapley Additive exPlanations) explainable AI technique, allowing us to identify the specific features that underlie the accurate detection of attacks.

Motivation & Objective

• Address the limited generalizability of existing behavior-based IoT intrusion detection models due to data leakage and overfitting.
• Develop a reproducible and robust methodology for IoT attack detection using isolated train/test splits to prevent data leakage.
• Improve detection performance through an enhanced rolling window feature extraction method and a genetic algorithm-based multi-step feature selection process.
• Ensure model interpretability and confidence via explainable AI (SHAP) to identify key features underlying attack detection.
• Validate the framework across multiple datasets, ML models, and evaluation metrics to ensure reliability and generalization.

Proposed method

• Implement a novel rolling window approach for feature extraction that captures temporal network behavior more effectively than conventional methods.
• Apply a multi-step feature selection process using a genetic algorithm with exogenous feedback to eliminate redundant or overfitting-prone features.
• Use strictly isolated training and testing datasets to prevent data leakage and ensure model generalizability.
• Evaluate the framework across a diverse portfolio of machine learning models (e.g., SVM, XGBoost, Random Forest, Logistic Regression) and evaluation metrics.
• Employ SHAP (SHapley Additive exPlanations) to interpret model decisions and identify the most influential features for attack detection.
• Make all code and datasets publicly available via GitHub to ensure reproducibility and community validation.

Figure 2: The visualisation of steps in system implementation.

Experimental results

Research questions

• RQ1 Can a behavior-based ML model for IoT attack detection achieve high generalizability across unseen datasets and attack types?
• RQ2 How does the proposed rolling window feature extraction method improve detection performance compared to conventional approaches?
• RQ3 To what extent does the multi-step feature selection process reduce overfitting and improve model robustness?
• RQ4 Which network features are most predictive of specific IoT attacks, and how can their importance be quantitatively assessed?
• RQ5 Can explainable AI techniques (SHAP) effectively identify and validate the features driving accurate attack detection?

Key findings

• The IoTGeM framework achieved an F1-score of 0.994 on the PS (Port Scanning) dataset using Logistic Regression, demonstrating near-perfect detection performance.
• On the SYN (SYN Flood) dataset, XGBoost achieved an F1-score of 0.969, indicating strong generalization across diverse attack types.
• The use of isolated train/test splits significantly reduced data leakage risks, enhancing model reliability and generalizability.
• SHAP analysis revealed that features such as packet frequency, flow duration, and protocol-specific anomalies were consistently predictive across multiple attack types.
• The multi-step feature selection process reduced overfitting, as evidenced by consistent performance across different models and datasets.
• The framework outperformed baseline methods in terms of F1-score and AUC across all evaluated datasets, with minimal variance in performance metrics.

Figure 3: Use of data in this study. Train/CV=Training and cross-validation. HPO=Feature reduction and hyperparameter selection. Validation=Data validation and feature selection. S-n= Session n.