[Paper Review] End-to-end Interpretable Neural Motion Planner

Wenyuan Zeng, Wenjie Luo+5 more|arXiv (Cornell University)|Jan 17, 2021

Autonomous Vehicle Technology and Safety32 references4 citations

TL;DR

This paper proposes an end-to-end interpretable neural motion planner (NMP) that processes raw LiDAR and HD map data to generate 3D detections, future trajectories, and a time-dependent cost volume for motion planning. The model samples diverse, physically feasible trajectories and selects the one with minimal learned cost, achieving safer planning than baselines while maintaining state-of-the-art performance in detection and motion forecasting on real-world urban driving data.

ABSTRACT

In this paper, we propose a neural motion planner (NMP) for learning to drive autonomously in complex urban scenarios that include traffic-light handling, yielding, and interactions with multiple road-users. Towards this goal, we design a holistic model that takes as input raw LIDAR data and a HD map and produces interpretable intermediate representations in the form of 3D detections and their future trajectories, as well as a cost volume defining the goodness of each position that the self-driving car can take within the planning horizon. We then sample a set of diverse physically possible trajectories and choose the one with the minimum learned cost. Importantly, our cost volume is able to naturally capture multi-modality. We demonstrate the effectiveness of our approach in real-world driving data captured in several cities in North America. Our experiments show that the learned cost volume can generate safer planning than all the baselines.

Motivation & Objective

• To bridge the gap between end-to-end learning and traditional modular autonomous driving stacks by enabling interpretable, jointly optimized perception and planning.
• To develop a motion planner that produces interpretable intermediate outputs—3D detections, future trajectories, and a space-time cost volume—while learning from real-world data.
• To improve safety and robustness in complex urban scenarios, including traffic-light handling, yielding, and multi-agent interactions.
• To enable better interpretability and failure analysis by exposing intermediate representations such as cost maps and motion forecasts.
• To outperform both end-to-end and modular baselines in planning safety, detection accuracy, and motion forecasting performance.

Proposed method

• The model uses a backbone network to process raw LiDAR point clouds and HD maps, producing 3D bounding boxes for surrounding actors and their predicted future trajectories.
• It generates a multi-filter cost volume that encodes the 'goodness' of each possible future position for the ego-vehicle across the planning horizon, enabling multi-modality capture.
• The cost volume is learned end-to-end via a joint optimization objective combining detection, motion forecasting, and planning losses.
• A trajectory sampler generates diverse, physically feasible candidate trajectories based on the cost volume.
• The final motion plan is selected as the trajectory with the lowest cumulative cost across all time steps, indexed from the cost volume filters.
• The model incorporates a traffic rule penalty to discourage unsafe behaviors such as collisions and lane violations.

Figure 1: Our end-to-end interpretable neural motion planner (NMP). Backbone network takes LiDAR data and maps as inputs, and outputs bounding boxes of other actors for future timesteps (perception), as well as a cost volume for planning with $T$ filters. Next, for each trajectory proposal from the

Experimental results

Research questions

• RQ1 Can an end-to-end neural network jointly optimize perception and motion planning while maintaining interpretability through intermediate representations?
• RQ2 How well can a learned cost volume capture multi-modal driving behaviors such as lane changing versus lane keeping?
• RQ3 Does joint optimization of detection, forecasting, and planning lead to improved safety and performance compared to task-specific or modular systems?
• RQ4 To what extent can the model generalize to complex urban scenarios with heavy traffic and dynamic interactions?
• RQ5 Can the learned cost volume outperform manually designed cost functions in real-world driving scenarios?

Key findings

• The proposed NMP achieves state-of-the-art mAP of 94.2 at IoU 0.5, 90.8 at 0.6, and 81.1 at 0.7, outperforming specialized detectors like Pixor and IntentNet in 3D detection.
• For motion forecasting, the model achieves lower L1 and L2 errors than baselines across all timesteps, with L2 error of 0.85m at t=5s, showing improved trajectory prediction accuracy.
• The model reduces future collision rates by a large margin—over 50% lower than IL and Ego-motion baselines—demonstrating superior safety in planning.
• The ablation study confirms that joint training with planning loss and traffic rule penalty significantly improves planning performance, reducing collision and lane violation rates.
• Visualization of the cost map shows that the model learns time-dependent, scenario-aware cost distributions, including multi-modality, lane-keeping, and collision avoidance behaviors.
• The model achieves lower lane violation rates than ACC (which is designed to follow lanes), indicating it learns map-aware behavior from data rather than relying on hardcoded rules.

Figure 2: Trajectory Representation. We first sample a set of parameters of a Clothoid to determine the shape of a trajectory. We then sample a velocity profile to determine how fast the SDV go along this trajectory. Combining these two, we can get a space-time trajectory.