[Paper Review] An LSTM Network for Highway Trajectory Prediction

Florent Altché, Arnaud de La Fortelle|arXiv (Cornell University)|Jan 24, 2018

Autonomous Vehicle Technology and Safety29 references635 citations

TL;DR

This paper proposes an LSTM-based neural network for predicting highway vehicle trajectories using the full NGSIM US101 dataset, achieving a median RMS error of 0.73 m laterally and 2.96 m/s longitudinally for 10-second forecasts. The model leverages temporal dependencies in naturalistic driving data to improve long-term prediction accuracy over prior methods, which often rely on biased or limited trajectory collections.

ABSTRACT

In order to drive safely and efficiently on public roads, autonomous vehicles will have to understand the intentions of surrounding vehicles, and adapt their own behavior accordingly. If experienced human drivers are generally good at inferring other vehicles' motion up to a few seconds in the future, most current Advanced Driving Assistance Systems (ADAS) are unable to perform such medium-term forecasts, and are usually limited to high-likelihood situations such as emergency braking. In this article, we present a first step towards consistent trajectory prediction by introducing a long short-term memory (LSTM) neural network, which is capable of accurately predicting future longitudinal and lateral trajectories for vehicles on highway. Unlike previous work focusing on a low number of trajectories collected from a few drivers, our network was trained and validated on the NGSIM US-101 dataset, which contains a total of 800 hours of recorded trajectories in various traffic densities, representing more than 6000 individual drivers.

Motivation & Objective

• Address the lack of consistent medium-term trajectory prediction in current ADAS systems, which often fail to anticipate complex maneuvers beyond emergency braking.
• Overcome the limitations of prior work that uses handpicked or small-scale trajectory datasets, which introduce selection bias and reduce real-world generalization.
• Develop a deep learning model capable of accurately forecasting both longitudinal and lateral vehicle trajectories over extended horizons (up to 10 seconds) in diverse traffic conditions.
• Improve motion planning for autonomous vehicles by enabling more reliable prediction of surrounding vehicle behavior, especially during overtaking and lane changes.
• Validate the model on a large-scale, real-world dataset (NGSIM US101) to ensure robustness and generalization to typical highway driving scenarios.

Proposed method

• The model employs a long short-term memory (LSTM) neural network architecture to learn temporal dependencies from sequential vehicle state data.
• Input features include longitudinal and lateral positions, velocities, and relative positions of surrounding vehicles within a defined range, extracted from the NGSIM US101 dataset.
• Data preprocessing involves normalizing features and constructing fixed-length temporal windows (Kprev = 10) of past observations to feed into the network.
• Two separate output heads predict future lateral position and longitudinal velocity, trained jointly using mean squared error loss.
• A bagging technique is applied by averaging predictions from four best-performing models to improve robustness and reduce variance.
• Training is performed on the entire US101 dataset (800 hours, 6000+ drivers) without prior trajectory selection, ensuring broad representativeness of real-world driving behavior.

Experimental results

Research questions

• RQ1 How well can an LSTM network generalize to long-term trajectory prediction (up to 10 seconds) when trained on a large, unbiased dataset of naturalistic highway driving?
• RQ2 What is the impact of including vehicle type information on prediction accuracy for lateral and longitudinal motion?
• RQ3 How does the model perform in predicting rare but critical events such as lane changes or sudden braking, compared to standard regression metrics?
• RQ4 Can the use of ensemble learning (bagging) improve prediction stability and reduce error compared to individual models?
• RQ5 Does the model exhibit delayed responses in prediction, particularly during transitions like lane changes, and how does this affect its practical utility?

Key findings

• The proposed LSTM model achieves a median RMS error of 0.73 m for lateral position prediction and 2.96 m/s for longitudinal velocity when forecasting 10 seconds ahead.
• Using the bagged ensemble of four models yields the best performance, reducing lateral error by 12% compared to the reference model at the 10-second horizon.
• Prediction accuracy degrades gradually with increasing forecast horizon, with lateral error reaching 0.73 m and longitudinal velocity error 2.96 m/s at 10 seconds.
• Despite strong overall performance, the model exhibits a noticeable delay in responding to behavioral changes—up to 8–9 seconds for lateral position—especially at longer horizons.
• Providing vehicle type information improves longitudinal speed prediction (reducing error by ~0.2 m/s on average) but offers no benefit for lateral position forecasting.
• The model outperforms prior state-of-the-art methods, including a multi-layer perceptron baseline from [14], particularly at longer horizons, demonstrating the advantage of recurrent memory in trajectory modeling.