[Paper Review] One Thousand and One Hours: Self-driving Motion Prediction Dataset

John A. Houston, Guido Zuidhof+7 more|arXiv (Cornell University)|Jun 25, 2020

Autonomous Vehicle Technology and Safety28 references200 citations

TL;DR

The paper releases the largest self-driving motion prediction dataset to date, with 1,118 hours over 170,000 scenes, plus a detailed HD semantic map and aerial imagery, plus L5Kit tooling and baseline models for motion forecasting and planning.

ABSTRACT

Motivated by the impact of large-scale datasets on ML systems we present the largest self-driving dataset for motion prediction to date, containing over 1,000 hours of data. This was collected by a fleet of 20 autonomous vehicles along a fixed route in Palo Alto, California, over a four-month period. It consists of 170,000 scenes, where each scene is 25 seconds long and captures the perception output of the self-driving system, which encodes the precise positions and motions of nearby vehicles, cyclists, and pedestrians over time. On top of this, the dataset contains a high-definition semantic map with 15,242 labelled elements and a high-definition aerial view over the area. We show that using a dataset of this size dramatically improves performance for key self-driving problems. Combined with the provided software kit, this collection forms the largest and most detailed dataset to date for the development of self-driving machine learning tasks, such as motion forecasting, motion planning and simulation. The full dataset is available at http://level5.lyft.com/.

Motivation & Objective

• Highlight the impact of large-scale data on motion forecasting and planning for autonomous driving.
• Provide a detailed, route-specific dataset with rich semantic maps and imagery to enable downstream ML tasks.
• Offer open-source tooling (L5Kit) and baseline models to facilitate research and benchmarking.

Proposed method

• Curate a large-scale dataset of 170,000 scenes (25 s each, 10 Hz) from 20 vehicles on a fixed Palo Alto route.
• Provide a high-definition semantic map with 15,242 annotations and 8,505 lane segments plus a 74 km^2 aerial map.
• Release L5Kit, a Python toolkit for data access, visualization, and baselines for motion forecasting and planning.
• Establish baseline motion forecasting using BEV rasters and a ResNet-50 backbone with history information.
• Demonstrate a planning baseline by training a model to predict and execute 5-second horizon trajectories with perturbations to mitigate compounding errors.

Experimental results

Research questions

• RQ1 How does scale (hours, scenes, and participants) affect motion forecasting accuracy and planning performance in self-driving systems?
• RQ2 What value do detailed HD semantic maps and aerial context provide for predicting future actor motions?
• RQ3 Can open ML-based planning be effectively evaluated on open datasets with closed-loop simulations?
• RQ4 How do baseline models trained on this dataset compare to perception-centric baselines for forecasting and planning?

Key findings

• The dataset constitutes 1,118 hours, 26,344 km, and 162k scenes across 20 vehicles.
• HD semantic map includes 15,242 annotations and 8,505 lane segments, plus high-resolution aerial imagery over 74 km^2.
• Motion forecasting baseline shows improved displacement/error metrics as training data increases and benefits from history information.
• Planning baseline demonstrates improved closed-loop performance with larger training data and synthetic perturbations to address compounding errors.
• The dataset and L5Kit enable open evaluation of ML-based planning, previously addressed mainly with proprietary data.
• Both forecasting and planning performance appear non-saturated with dataset size, suggesting further gains with tens of thousands to millions of hours.