[Paper Review] A2D2: Audi Autonomous Driving Dataset

Jakob Geyer, Yohannes Kassahun+17 more|arXiv (Cornell University)|Apr 14, 2020

Remote Sensing and LiDAR ApplicationsEnvironmental Science28 references265 citations

TL;DR

A2D2 provides a commercially usable, fully synchronized multi-modal dataset (6 cameras, 5 LiDARs) with 360-degree coverage, including semantic/instance segmentation and 3D bounding boxes, plus extensive vehicle bus data for autonomous driving research under CC BY-ND 4.0. It includes labeled and unlabeled sequences from Germany and a tutorial for access.

ABSTRACT

Research in machine learning, mobile robotics, and autonomous driving is accelerated by the availability of high quality annotated data. To this end, we release the Audi Autonomous Driving Dataset (A2D2). Our dataset consists of simultaneously recorded images and 3D point clouds, together with 3D bounding boxes, semantic segmentation, instance segmentation, and data extracted from the automotive bus. Our sensor suite consists of six cameras and five LiDAR units, providing full 360 degree coverage. The recorded data is time synchronized and mutually registered. Annotations are for non-sequential frames: 41,277 frames with semantic segmentation image and point cloud labels, of which 12,497 frames also have 3D bounding box annotations for objects within the field of view of the front camera. In addition, we provide 392,556 sequential frames of unannotated sensor data for recordings in three cities in the south of Germany. These sequences contain several loops. Faces and vehicle number plates are blurred due to GDPR legislation and to preserve anonymity. A2D2 is made available under the CC BY-ND 4.0 license, permitting commercial use subject to the terms of the license. Data and further information are available at http://www.a2d2.audi.

Motivation & Objective

• Catalyze autonomous driving research by providing a commercially usable, richly annotated dataset.
• Offer full-surround camera and LiDAR data synchronized to a global frame.
• Include extensive vehicle bus data to enable end-to-end and reinforcement learning research.
• Provide anonymized data and tutorials to ease community adoption.
• Enable benchmarks and challenges to compare perception algorithms across modalities.

Proposed method

• Data collection on an Audi Q7 e-tron with six cameras and five Velodyne VLP-16 LiDARs.
• Rigorous sensor calibration and registration to a common global reference frame.
• Annotation of semantic and instance segmentation for 41,277 frames across 38 classes.
• 3D bounding boxes provided for 12,497 front-camera frames within the front-center camera FOV.
• Release of 392,556 unannotated sequences for self-supervised or SLAM research.
• Baseline semantic segmentation experiments using a ResNet-101 encoder with PSP-Net decoder.

Experimental results

Research questions

• RQ1 How can a multimodal, surround-sensor automotive dataset support development of perception and SLAM algorithms?
• RQ2 What is the impact of using pre-trained weights and anonymization on semantic segmentation performance on A2D2?
• RQ3 Does the dataset's inclusion of vehicle bus data enable broader research beyond object detection (e.g., end-to-end or reinforcement learning)?

Key findings

• The dataset provides 41,277 semantically/instance-labeled images and 12,497 frames with 3D bounding boxes in front-camera FOV.
• Five LiDARs and six cameras yield full 360-degree coverage with time-synchronized, registered data.
• A semantic segmentation model achieves a mean IoU of 71.01% on 18 foreground classes (baseline ResNet-101 + PSP-Net).
• Using ImageNet-pretrained weights improves mean IoU to 71.01% (baseline) versus alternatives (68.40% with ResNet-50 + PSP-Net, 65.31% without pre-training).
• Anonymization (face/plate blurring) has minimal impact on mean IoU (70.94%) compared to non-anonymized baseline (71.01%).
• The dataset supports end-to-end and self-supervised learning through unannotated sequences and vehicle bus data.