[Paper Review] Externally validating the IoTDevID device identification methodology using the CIC IoT 2022 Dataset

Kahraman Kostas, Mike Just+1 more|arXiv (Cornell University)|Jul 3, 2023

Internet Traffic Analysis and Secure E-votingComputer Science3 citations

TL;DR

This paper externally validates the IoTDevID device identification methodology using the CIC-IoT-2022 dataset, demonstrating that its feature set and aggregation algorithm achieve strong performance—92.50 F1 score on 31 IP-only device classes—while highlighting the critical role of data diversity and quantity in model success, especially for non-IP devices with limited data.

ABSTRACT

In the era of rapid IoT device proliferation, recognizing, diagnosing, and securing these devices are crucial tasks. The IoTDevID method (IEEE Internet of Things 2022) proposes a machine learning approach for device identification using network packet features. In this article we present a validation study of the IoTDevID method by testing core components, namely its feature set and its aggregation algorithm, on a new dataset. The new dataset (CIC-IoT-2022) offers several advantages over earlier datasets, including a larger number of devices, multiple instances of the same device, both IP and non-IP device data, normal (benign) usage data, and diverse usage profiles, such as active and idle states. Using this independent dataset, we explore the validity of IoTDevID's core components, and also examine the impacts of the new data on model performance. Our results indicate that data diversity is important to model performance. For example, models trained with active usage data outperformed those trained with idle usage data, and multiple usage data similarly improved performance. Results for IoTDevID were strong with a 92.50 F1 score for 31 IP-only device classes, similar to our results on previous datasets. In all cases, the IoTDevID aggregation algorithm improved model performance. For non-IP devices we obtained a 78.80 F1 score for 40 device classes, though with much less data, confirming that data quantity is also important to model performance.

Motivation & Objective

• To externally validate the IoTDevID methodology using a new, more comprehensive dataset to assess its robustness and generalizability.
• To evaluate the impact of data diversity—such as active vs. idle usage states and multiple device instances—on model performance.
• To examine the effectiveness of IoTDevID’s feature selection and aggregation algorithm in real-world conditions with heterogeneous device types.
• To investigate the influence of data quantity and quality on identification accuracy, particularly for non-IP devices.
• To ensure methodological rigor by avoiding data leakage and using isolated training/testing splits to enhance reproducibility.

Proposed method

• The study applies the IoTDevID framework to the CIC-IoT-2022 dataset, which includes IP and non-IP devices (e.g., ZigBee, Z-Wave), with both active and idle usage profiles.
• A multi-layer feature selection process using a genetic algorithm eliminates overfitting-prone features such as session-specific identifiers (e.g., port numbers, sequence numbers).
• The core aggregation algorithm combines similarly modeled packets to improve identification accuracy by reducing noise and enhancing feature consistency.
• Models are trained and tested on isolated data splits to prevent data leakage, ensuring valid performance estimation.
• Performance is evaluated using standard metrics: precision, recall, and F1 score, with macro and weighted averages reported.
• Experiments compare model performance across different data conditions: active usage, idle usage, and combined usage profiles.

Figure 1: Example transfer problem in the Aalto dataset. Network data is collected at the gateway. Between D-LinkDoorSensor and D-LinkHomeHub there is only Z-Wave as a communication medium. Between D-LinkHomeHub and the gateway, there is only Ethernet. Data from the D-LinkDoorSensor is decapsulated

Experimental results

Research questions

• RQ1 Does the IoTDevID methodology generalize well to a new, larger, and more diverse dataset like CIC-IoT-2022?
• RQ2 How does data diversity—specifically active vs. idle device usage—impact the performance of device identification models?
• RQ3 To what extent does the IoTDevID aggregation algorithm improve identification accuracy across different device types?
• RQ4 How do data quantity and quality affect model performance, particularly for non-IP devices with limited training samples?
• RQ5 Can the IoTDevID framework maintain high performance when tested on real-world data with heterogeneous device characteristics and usage profiles?

Key findings

• The IoTDevID method achieved a 92.50 F1 score on 31 IP-only device classes when evaluated on active usage data, confirming strong generalization and robustness.
• Models trained on active usage data outperformed those trained on idle usage data, indicating that data diversity significantly enhances model performance.
• The IoTDevID aggregation algorithm consistently improved performance for IP devices, but had no effect on non-IP devices due to shared MAC addresses triggering exception handling.
• For non-IP devices, a 78.80 F1 score was achieved across 40 device classes, though performance was limited by insufficient training data, as shown by a negative correlation between packet count and F1 score.
• The inclusion of non-IP devices did not degrade performance on IP devices, with only minor exceptions (e.g., Sonos One Speaker), suggesting the method’s stability across device types.
• The study confirms that data leakage and feature overfitting are avoidable in IoT device identification when proper data splitting and feature selection techniques are applied.

Figure 2: Steps of the IoTDevID study.