[Paper Review] VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation

Jiyang Gao, Chen Sun+5 more|arXiv (Cornell University)|May 8, 2020

Autonomous Vehicle Technology and Safety29 references44 citations

TL;DR

VectorNet uses a hierarchical graph neural network to encode vectorized HD maps and agent trajectories for behavior prediction, achieving competitive or superior results with far fewer parameters and FLOPs than rasterized ConvNet baselines, and state-of-the-art on Argoverse.

ABSTRACT

Behavior prediction in dynamic, multi-agent systems is an important problem in the context of self-driving cars, due to the complex representations and interactions of road components, including moving agents (e.g. pedestrians and vehicles) and road context information (e.g. lanes, traffic lights). This paper introduces VectorNet, a hierarchical graph neural network that first exploits the spatial locality of individual road components represented by vectors and then models the high-order interactions among all components. In contrast to most recent approaches, which render trajectories of moving agents and road context information as bird-eye images and encode them with convolutional neural networks (ConvNets), our approach operates on a vector representation. By operating on the vectorized high definition (HD) maps and agent trajectories, we avoid lossy rendering and computationally intensive ConvNet encoding steps. To further boost VectorNet's capability in learning context features, we propose a novel auxiliary task to recover the randomly masked out map entities and agent trajectories based on their context. We evaluate VectorNet on our in-house behavior prediction benchmark and the recently released Argoverse forecasting dataset. Our method achieves on par or better performance than the competitive rendering approach on both benchmarks while saving over 70% of the model parameters with an order of magnitude reduction in FLOPs. It also outperforms the state of the art on the Argoverse dataset.

Motivation & Objective

• Integrate agent dynamics with structured HD map context using a vectorized representation.
• Develop a hierarchical graph architecture to capture local polyline interactions and global polyline-node relations.
• Introduce a self-supervised graph completion objective to improve context learning.
• Evaluate on in-house and Argoverse datasets to compare with rasterized rendering baselines and state-of-the-art.

Proposed method

• Represent maps and agent trajectories as sequences of vectors (polylines) and treat each vector as a graph node with features including start/end coordinates and attributes.
• Construct polyline subgraphs by connecting vectors within the same polyline and aggregating to polyline features via a local GNN (MLP-based) and max-pooling.
• Model high-order interactions with a global graph using a self-attention based GNN over polyline features; decode target agent futures from the corresponding polyline node features.
• Introduce a graph completion auxiliary task by masking node features and training the network to reconstruct them, encouraging context-aware representations.
• Optimize a multi-task loss: L = L_traj (negative Gaussian log-likelihood for future trajectories) + α L_node (Huber loss for masked node feature reconstruction).
• Normalize polyline node features to unit scale and rotate coordinates to align with the target vehicle heading for stable prediction.

Experimental results

Research questions

• RQ1 Can a vectorized representation of HD maps and agent trajectories be effectively learned with a hierarchical graph to predict future behavior?
• RQ2 Does a locally-connected polyline subgraph plus a global attention-based graph improve trajectory prediction compared to rasterized ConvNet baselines?
• RQ3 Does a graph completion auxiliary task improve the modeling of interactions between agents and map context?

Key findings

• VectorNet achieves on-par or better performance than rasterized ConvNet baselines while using substantially fewer parameters (~72K vs ~246K) and an order of magnitude fewer FLOPs.
• VectorNet shows state-of-the-art DE@3s on the Argoverse test set and outperforms the best ConvNet baselines on Argoverse by a notable margin.
• On the in-house dataset, VectorNet with vectorized inputs matches or exceeds the rasterized baseline performance with a fraction of compute (FLOPs) and parameters.
• Ablations reveal that incorporating both map polylines and other agents’ trajectories improves accuracy, and the graph completion auxiliary task consistently helps, especially at longer horizons.
• The polyline subgraph depth (three layers) and a single-layer global graph provide best trade-offs between accuracy and efficiency.