[Paper Review] LiRaNet: End-to-End Trajectory Prediction using Spatio-Temporal Radar Fusion

Meet Shah, Zhiling Huang+6 more|arXiv (Cornell University)|Oct 1, 2020

Autonomous Vehicle Technology and SafetyEngineering17 citations

TL;DR

LiRaNet proposes an end-to-end trajectory prediction model that fuses spatio-temporal radar data with lidar and HD maps to improve motion forecasting accuracy. By leveraging radar's radial velocity and long-range detection, the method achieves a 52% reduction in prediction error for high-acceleration objects and a 16% reduction at longer ranges, outperforming lidar-only baselines in safety-critical scenarios with sparse lidar data.

ABSTRACT

In this paper, we present LiRaNet, a novel end-to-end trajectory prediction method which utilizes radar sensor information along with widely used lidar and high definition (HD) maps. Automotive radar provides rich, complementary information, allowing for longer range vehicle detection as well as instantaneous radial velocity measurements. However, there are factors that make the fusion of lidar and radar information challenging, such as the relatively low angular resolution of radar measurements, their sparsity and the lack of exact time synchronization with lidar. To overcome these challenges, we propose an efficient spatio-temporal radar feature extraction scheme which achieves state-of-the-art performance on multiple large-scale datasets.Further, by incorporating radar information, we show a 52% reduction in prediction error for objects with high acceleration and a 16% reduction in prediction error for objects at longer range.

Motivation & Objective

• To address the limitations of lidar-only trajectory prediction in scenarios with sparse points, high acceleration, or long-range detection.
• To overcome challenges in radar fusion, including low angular resolution, sparsity, positional uncertainty, and lack of time synchronization with lidar.
• To enable early fusion of radar with lidar and HD maps for improved motion forecasting in autonomous driving.
• To demonstrate state-of-the-art performance on large-scale datasets while maintaining low inference latency.

Proposed method

• The model uses a graph-based spatio-temporal feature extraction scheme to fuse sequences of radar point clouds into dynamic, BEV-grid representations.
• Radar features are extracted per grid cell using k-nearest neighbor search and processed through a learnable module to encode spatio-temporal dynamics.
• Lidar and HD map features are fused early with radar features at the feature level, enabling joint modeling of geometric and dynamic information.
• The network employs a transformer-based backbone to aggregate multi-sensor features and predict future trajectories end-to-end.
• Runtime optimization is achieved via CUDA-optimized kernels for per-cell radar processing and TensorRT inference acceleration.
• Temporal fusion layers are designed to handle variable-length radar history (up to 0.5s) with minimal latency impact.

Experimental results

Research questions

• RQ1 Can early fusion of radar with lidar and HD maps improve trajectory prediction performance in complex, real-world driving scenarios?
• RQ2 How does radar-based velocity estimation enhance prediction accuracy for high-acceleration or long-range objects where lidar is sparse?
• RQ3 To what extent does radar improve mode ambiguity resolution in multi-modal trajectory prediction?
• RQ4 What is the latency overhead of incorporating radar data into an end-to-end trajectory prediction system?
• RQ5 Can radar improve detection performance in occluded or low-point-density scenarios?

Key findings

• LiRaNet achieves a 52% reduction in trajectory prediction error for objects with high acceleration compared to lidar-only baselines.
• A 16% reduction in prediction error is observed for objects at longer ranges, where lidar point density is low.
• The model improves detection performance, with a 7.57-point increase in AP0.5 on nuScenes and a 2.30-point increase on X17k when radar is added.
• Runtime impact of adding radar is minimal, with only a 3.5ms increase in latency (from 38.02ms to 40.98ms) even with 0.5s of radar history.
• Qualitative results show radar significantly improves prediction in occlusion, high-speed, and low-point-density scenarios.
• The addition of radar resolves mode ambiguity in multi-modal predictions, reducing incorrect trajectory hypotheses such as unsafe lane crossings or abrupt turns.