[Paper Review] Argoverse: 3D Tracking and Forecasting with Rich Maps

Ming-Fang Chang, John Lambert+9 more|arXiv (Cornell University)|Nov 6, 2019

Autonomous Vehicle Technology and Safety53 references157 citations

TL;DR

Argoverse introduces two large-scale autonomous driving datasets—3D Tracking with synchronized LiDAR and 360° stereo imagery plus rich HD maps, and Motion Forecasting with mined trajectories—demonstrating that HD map context improves 3D tracking and forecasting performance.

ABSTRACT

We present Argoverse -- two datasets designed to support autonomous vehicle machine learning tasks such as 3D tracking and motion forecasting. Argoverse was collected by a fleet of autonomous vehicles in Pittsburgh and Miami. The Argoverse 3D Tracking dataset includes 360 degree images from 7 cameras with overlapping fields of view, 3D point clouds from long range LiDAR, 6-DOF pose, and 3D track annotations. Notably, it is the only modern AV dataset that provides forward-facing stereo imagery. The Argoverse Motion Forecasting dataset includes more than 300,000 5-second tracked scenarios with a particular vehicle identified for trajectory forecasting. Argoverse is the first autonomous vehicle dataset to include "HD maps" with 290 km of mapped lanes with geometric and semantic metadata. All data is released under a Creative Commons license at www.argoverse.org. In our baseline experiments, we illustrate how detailed map information such as lane direction, driveable area, and ground height improves the accuracy of 3D object tracking and motion forecasting. Our tracking and forecasting experiments represent only an initial exploration of the use of rich maps in robotic perception. We hope that Argoverse will enable the research community to explore these problems in greater depth.

Motivation & Objective

• Provide large-scale, multimodal data for 3D object tracking and motion forecasting in autonomous driving.
• Introduce HD map components (vector lane centerlines, ground height, driveable area) and demonstrate their utility in perception tasks.
• Offer ground-truth 3D track annotations across diverse classes and a forecasting benchmark with diverse, realistic scenarios.

Proposed method

• Release two datasets (3D Tracking and Motion Forecasting) with synchronized LiDAR, 360° RGB video, front stereo, and 6-DoF localization.
• Provide HD maps including a vector lane graph, raster ground height, and raster driveable area/ROI.
• Annotate 3D cuboid tracks for 15 object classes within 5 m of driveable area.
• Mine 5 s future trajectories from 1006 hours of data to create a large forecasting benchmark (324,557 sequences).
• Offer an API connecting map data with sensor data to enable map-based perception and forecasting baselines.

Experimental results

Research questions

• RQ1 How can HD map information (lane centerlines, driveable area, ground height) improve 3D object tracking in autonomous driving?
• RQ2 How can rich semantic maps be leveraged to improve motion forecasting in complex driving scenarios (intersections, merges, dense traffic)?
• RQ3 What is the impact of map-based priors and multimodal predictions on tracking accuracy and forecasting diversity?
• RQ4 How does map context influence ground removal, orientation snapping, and trajectory pruning in forecasting models?

Key findings

• HD map context improves 3D tracking accuracy, particularly ground removal and orientation alignment to lane direction.
• Forecasting benchmarks show that map-derived centerlines and driveable-area priors enable diverse, plausible future trajectories with higher drivable-area compliance (DAC).
• Baseline forecasting models incorporating social context and map features (centerlines, lane offsets) achieve better minADE/minFDE and DAC across multiple prediction horizons (up to 3 s).
• The dataset contains 113 log segments for tracking (11,052 annotated objects) and 324,557 five-second forecasting sequences, enabling rich evaluation of multimodal predictions.
• Larger, richer maps enable map-automation research and provide a benchmark for HD-map-driven perception and forecasting methods.