[Paper Review] A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook

Mingyu Liu, Ekim Yurtsever+6 more|arXiv (Cornell University)|Jan 2, 2024

Data Stream Mining Techniques4 citations

TL;DR

This paper presents a comprehensive survey of over 200 autonomous driving datasets, analyzing their sensor modalities, data statistics, annotation quality, and task diversity. It introduces an impact score metric to evaluate dataset influence and identifies emerging trends such as end-to-end driving, vision-language modeling, and VLM-driven data generation for future datasets.

ABSTRACT

Autonomous driving has rapidly developed and shown promising performance due to recent advances in hardware and deep learning techniques. High-quality datasets are fundamental for developing reliable autonomous driving algorithms. Previous dataset surveys either focused on a limited number or lacked detailed investigation of dataset characteristics. To this end, we present an exhaustive study of 265 autonomous driving datasets from multiple perspectives, including sensor modalities, data size, tasks, and contextual conditions. We introduce a novel metric to evaluate the impact of datasets, which can also be a guide for creating new datasets. Besides, we analyze the annotation processes, existing labeling tools, and the annotation quality of datasets, showing the importance of establishing a standard annotation pipeline. On the other hand, we thoroughly analyze the impact of geographical and adversarial environmental conditions on the performance of autonomous driving systems. Moreover, we exhibit the data distribution of several vital datasets and discuss their pros and cons accordingly. Finally, we discuss the current challenges and the development trend of the future autonomous driving datasets.

Motivation & Objective

• To provide a comprehensive, systematic analysis of over 200 autonomous driving datasets across sensor modalities, data size, tasks, and contextual conditions.
• To introduce a novel impact score metric for evaluating the influence and importance of perception datasets in the research community.
• To analyze annotation methodologies and quality factors across major datasets, identifying key challenges and best practices.
• To investigate data distribution patterns across geography, time, and sensor configurations to inform dataset selection and development.
• To project future trends in autonomous driving datasets, including end-to-end learning, vision-language integration, and VLM-driven data generation.

Proposed method

• The authors compiled and analyzed 204 publicly available autonomous driving datasets from 2009 to 2023, covering real-world, synthetic, and hybrid data sources.
• They introduced an impact score metric based on citation frequency, dataset size, task coverage, and annotation quality to rank dataset influence.
• A detailed comparative analysis was conducted on 12 high-impact datasets across perception, prediction, planning, and control tasks.
• Annotation quality was evaluated through a framework assessing labeling consistency, bounding box density, and semantic precision across datasets like nuScenes, Waymo, and KITTI.
• Geographical and temporal distributions were visualized to reveal data bias and evolution trends in data collection.
• The study identifies emerging dataset trends, including end-to-end driving, multimodal language-vision data, and synthetic data generation via VLMs.

Experimental results

Research questions

• RQ1 What are the dominant sensor modalities, data sizes, and task distributions across the 200+ surveyed autonomous driving datasets?
• RQ2 How does the annotation quality vary across major datasets, and what factors most significantly affect it?
• RQ3 What is the impact of each dataset on the research community, and how can this be quantitatively measured?
• RQ4 How do data distributions vary across geography, time, and sensor configurations, and what biases do they reveal?
• RQ5 What are the key trends and challenges shaping the next generation of autonomous driving datasets?

Key findings

• The impact score metric successfully identifies high-influence datasets such as nuScenes, Waymo, and KITTI, which rank highly due to comprehensive annotation and broad task coverage.
• Annotation quality varies significantly: nuScenes and Waymo show superior labeling consistency and 3D bounding box precision, while some smaller datasets exhibit high noise and inconsistency.
• Real-world datasets dominate in perception tasks, but synthetic datasets are rapidly growing in use, especially for data augmentation and domain adaptation.
• End-to-end driving datasets remain scarce, with only 11 datasets identified, indicating a critical gap in training and evaluating unified driving agents.
• Vision-language datasets like DriveLM-nuScenes and LiDAR-text are emerging, supporting multimodal tasks such as visual question answering and 3D captioning.
• VLM-driven data generation, as demonstrated by DriveGAN and DriveDreamer, shows strong potential for creating high-fidelity, diverse synthetic data with minimal human annotation.