[Paper Review] ROAD: The Real ORNL Automotive Dynamometer Controller Area Network Intrusion Detection Dataset (with a comprehensive CAN IDS dataset survey & guide).

Miki E. Verma, Michael D. Iannacone+4 more|arXiv (Cornell University)|Dec 29, 2020

Vehicular Ad Hoc Networks (VANETs)Engineering30 references19 citations

TL;DR

This paper introduces the ROAD dataset, the first comprehensive CAN intrusion detection dataset featuring real, advanced attacks on a vehicle dynamometer, alongside a systematic survey of existing public CAN IDS datasets. It provides researchers with high-fidelity, labeled attack data and a detailed guide to selecting appropriate datasets for evaluating and comparing CAN intrusion detection systems.

ABSTRACT

The Controller Area Network (CAN) protocol is ubiquitous in modern vehicles, but the protocol lacks many important security properties, such as message authentication. To address these insecurities, a rapidly growing field of research has emerged that seeks to detect tampering, anomalies, or attacks on these networks; this field has developed a wide variety of novel approaches and algorithms to address these problems. One major impediment to the progression of this CAN anomaly detection and intrusion detection system (IDS) research area is the lack of high-fidelity datasets with realistic labeled attacks, without which it is difficult to evaluate, compare, and validate these proposed approaches. In this work we present the first comprehensive survey of publicly available CAN intrusion datasets. Based on a thorough analysis of the data and documentation, for each dataset we provide a detailed description and enumerate the drawbacks, benefits, and suggested use cases. Our analysis is aimed at guiding researchers in finding appropriate datasets for testing a CAN IDS. We present the Real ORNL Automotive Dynamometer (ROAD) CAN Intrusion Dataset, providing the first dataset with real, advanced attacks to the existing collection of open datasets.

Motivation & Objective

• Address the critical lack of high-fidelity, realistic, and labeled CAN intrusion datasets for evaluating and comparing intrusion detection systems (IDS) in automotive networks.
• Conduct a comprehensive survey of existing publicly available CAN IDS datasets to assess their quality, limitations, and suitability for research.
• Provide a detailed, evidence-based guide to help researchers select appropriate datasets based on their specific IDS evaluation needs.
• Present the Real ORNL Automotive Dynamometer (ROAD) dataset, featuring real-world, advanced attacks on a functional vehicle network under controlled conditions.
• Enable reproducible and valid evaluation of CAN IDS algorithms by offering a dataset with realistic attack scenarios and thorough documentation.

Proposed method

• Systematically collect and analyze all publicly available CAN intrusion detection datasets through a structured review process.
• Evaluate each dataset based on key criteria including data fidelity, attack realism, labeling accuracy, documentation quality, and reproducibility.
• Classify datasets by attack type, data collection environment, and intended use case to support informed selection by researchers.
• Design and execute real-world attack experiments on a full-scale automotive dynamometer at ORNL to generate the ROAD dataset.
• Record and label CAN traffic under normal operation and during a diverse set of advanced, realistic cyberattacks, including replay, spoofing, and denial-of-service attacks.
• Document all experimental conditions, attack vectors, and labeling procedures to ensure transparency and reproducibility for future research.

Experimental results

Research questions

• RQ1 What are the key limitations and strengths of existing publicly available CAN intrusion detection datasets?
• RQ2 How do different CAN IDS datasets compare in terms of data fidelity, attack realism, and documentation quality?
• RQ3 What types of attack scenarios are missing or underrepresented in current public datasets?
• RQ4 Can a real-world, high-fidelity dataset with advanced attacks improve the evaluation and validation of CAN IDS algorithms?
• RQ5 How can researchers be guided to select the most appropriate dataset for their specific IDS research objectives?

Key findings

• The ROAD dataset is the first publicly available CAN IDS dataset that captures real, advanced attacks on a functional vehicle network using a real automotive dynamometer.
• The survey reveals significant shortcomings in existing datasets, including low attack realism, poor labeling, and inadequate documentation, which hinder reliable evaluation of IDS approaches.
• Many existing datasets lack diverse attack types or fail to simulate realistic vehicle operating conditions, limiting their utility for robust IDS testing.
• The ROAD dataset includes a variety of advanced attacks such as replay, spoofing, and denial-of-service, executed under controlled but realistic conditions.
• The dataset is fully labeled and accompanied by detailed documentation, enabling reproducible experiments and fair comparison of IDS algorithms.
• The comprehensive survey provides a clear framework for dataset selection, helping researchers avoid common pitfalls in IDS evaluation.