[Paper Review] Scalability in Perception for Autonomous Driving: Waymo Open Dataset

Pei Sun, Henrik Kretzschmar+23 more|arXiv (Cornell University)|Dec 10, 2019

Video Surveillance and Tracking Methods21 references228 citations

TL;DR

Introduces a large-scale multimodal Waymo Open Dataset with synchronized LiDAR and camera data, extensive 3D/2D annotations, and baselines to study scalability and geographic generalization in autonomous driving perception.

ABSTRACT

The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the overall viability of the technology. In an effort to help align the research community's contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open.

Motivation & Objective

• Motivate and enable research on perception scalability for autonomous driving by releasing a large, diverse, multimodal dataset.
• Provide high-quality, exhaustively annotated LiDAR and camera data with consistent tracking IDs to support detection and tracking tasks.
• Examine how dataset size and geographic domain gaps affect 3D perception methods and generalization across geographies.

Proposed method

• Assemble a large multimodal dataset with synchronized LiDAR and camera data from multiple cities (urban/suburban) and provide precise calibration and synchronization data.
• Annotate 3D bounding boxes for LiDAR and 2D boxes for camera images with consistent track IDs across frames to support tracking.
• Define 2D and 3D detection and tracking benchmarks with specialized metrics, including APH (heading-aware AP) for 3D detection and MOT-based tracking metrics.
• Establish strong baselines using contemporary detectors (PointPillars for 3D LiDAR, Faster R-CNN for 2D) and a tracking-by-detection framework, plus domain gap analysis across cities.
• Offer dataset analysis tools and a rolling-shutter aware camera-LiDAR projection method to facilitate sensor fusion research.

Experimental results

Research questions

• RQ1 How does increasing dataset size impact performance for 3D LiDAR-based object detection and heading estimation?
• RQ2 What is the effect of geographic domain gaps (e.g., between different cities) on detector and tracker performance, and how can domain adaptation help?
• RQ3 Can joint sensor fusion (LiDAR + camera) and precise synchronization improve 3D and 2D perception benchmarks?
• RQ4 How does dataset diversity (urban vs suburban, time of day, weather indirectly through geography) affect generalization of perception models?
• RQ5 What baseline performance can be established for 2D and 3D object detection and tracking on this dataset?

Key findings

• The dataset enables pronounced domain gaps across geographies, offering opportunities for domain adaptation research.
• Baseline results demonstrate strong performance for both 3D LiDAR and 2D camera detections, as well as for tracking, with measurable differences across ranges and object classes.
• There is a clear benefit to larger, more diverse datasets for improving perception performance, especially for data-intensive models.
• Sensor synchronization and LiDAR range-image representations provide useful avenues for cross-domain learning and alternative input representations.
• Domain gap analyses show that training on one city and testing on another can significantly affect APH/AP metrics, highlighting the need for domain-aware training strategies.
• The dataset supports robust 3D and 2D tracking through consistent object IDs and multi-sensor fusion, enabling comprehensive MOT evaluation.