[Paper Review] VeReMi: A Dataset for Comparable Evaluation of Misbehavior Detection in VANETs

Rens W. van der Heijden, Thomas Lukaseder+1 more|arXiv (Cornell University)|Apr 18, 2018

Vehicular Ad Hoc Networks (VANETs)4 citations

TL;DR

VeReMi is a publicly available, extensible dataset for evaluating misbehavior detection in Vehicular Ad-hoc Networks (VANETs), enabling reproducible and comparable assessment of detection mechanisms. The study demonstrates that the acceptance range threshold and simple speed check are complementary, detecting different attack types, and advocates for fusion-based detection using VeReMi as a standardized evaluation baseline.

ABSTRACT

Vehicular networks are networks of communicating vehicles, a major enabling technology for future cooperative and autonomous driving technologies. The most important messages in these networks are broadcast-authenticated periodic one-hop beacons, used for safety and traffic efficiency applications such as collision avoidance and traffic jam detection. However, broadcast authenticity is not sufficient to guarantee message correctness. The goal of misbehavior detection is to analyze application data and knowledge about physical processes in these cyber-physical systems to detect incorrect messages, enabling local revocation of vehicles transmitting malicious messages. Comparative studies between detection mechanisms are rare due to the lack of a reference dataset. We take the first steps to address this challenge by introducing the Vehicular Reference Misbehavior Dataset (VeReMi) and a discussion of valid metrics for such an assessment. VeReMi is the first public extensible dataset, allowing anyone to reproduce the generation process, as well as contribute attacks and use the data to compare new detection mechanisms against existing ones. The result of our analysis shows that the acceptance range threshold and the simple speed check are complementary mechanisms that detect different attacks. This supports the intuitive notion that fusion can lead to better results with data, and we suggest that future work should focus on effective fusion with VeReMi as an evaluation baseline.

Motivation & Objective

• To address the lack of standardized, publicly available datasets for evaluating misbehavior detection in VANETs.
• To enable reproducible and comparable evaluation of detection mechanisms across different traffic scenarios and attack types.
• To establish a common baseline using simulation-generated data that supports independent validation and extension by the research community.
• To propose and evaluate new metrics for assessing detector performance, including error distribution and precision-recall analysis.
• To support future research in fusion-based detection and machine learning applications by providing a representative, extensible dataset.

Proposed method

• The VeReMi dataset is generated via large-scale, realistic VANET simulations using the Maat simulation framework, capturing vehicle dynamics, communication, and attack behaviors.
• The dataset includes periodic beacon messages with position, speed, and timestamp data, along with injected attacks such as false speed, false position, and sudden appearance attacks.
• Detection mechanisms are evaluated using precision-recall curves, which are identified as the preferred metric for comparative assessment in misbehavior detection.
• A novel metric based on the Gini index is introduced to analyze the spatial distribution of false positives, identifying localized weaknesses in detector performance.
• The framework supports plug-in detectors, enabling researchers to integrate and benchmark new detection algorithms against the same dataset.
• The dataset is designed to be extensible, allowing contributors to add new attack patterns and detection mechanisms.

Experimental results

Research questions

• RQ1 How can a standardized, publicly available dataset improve the comparability and reproducibility of misbehavior detection evaluations in VANETs?
• RQ2 Which detection mechanisms perform best under different attack types, and how do their error patterns vary spatially?
• RQ3 To what extent are the acceptance range threshold and simple speed check complementary in detecting distinct attack categories?
• RQ4 Can the spatial distribution of false positives, as measured by the Gini index, guide detector improvement or fusion strategies?
• RQ5 How suitable is VeReMi as a baseline for evaluating future detection mechanisms, including machine learning-based approaches?

Key findings

• The acceptance range threshold and simple speed check are complementary mechanisms, each detecting distinct types of attacks, indicating that fusion can improve overall detection performance.
• The spatial distribution of false positives, measured via the Gini index, reveals that errors are often concentrated in specific regions, suggesting opportunities for detector refinement or situational awareness integration.
• Precision-recall analysis is shown to be the most informative metric for comparing detection mechanisms, as it captures trade-offs between detection and false alarm rates.
• The VeReMi dataset enables consistent, reproducible evaluation across multiple detection mechanisms and supports future benchmarking and fusion research.
• The dataset is extensible and publicly available, allowing researchers to reproduce the generation process, contribute new attacks, and compare new detectors on equal footing.
• The study cautions that while VeReMi provides a strong baseline, its attack prevalence is not generalizable to all real-world conditions, especially under varying environmental or traffic parameters.