[Paper Review] Generic Probabilistic Interactive Situation Recognition and Prediction: From Virtual to Real

Jiachen Li, Hengbo Ma+2 more|arXiv (Cornell University)|Sep 9, 2018

Autonomous Vehicle Technology and Safety20 references4 citations

TL;DR

This paper proposes a generic, probabilistic hierarchical framework for interactive situation recognition and trajectory prediction in autonomous driving using a two-layer Hidden Markov Model (TLHMM) to model joint agent behavior and a Gaussian Mixture Model (GMM) for scene evolution. The method enables robust transfer learning from simulation to real-world data, reducing data and computational needs while achieving accurate, long-term joint trajectory predictions in complex scenarios like highway ramp merging.

ABSTRACT

Accurate and robust recognition and prediction of traffic situation plays an important role in autonomous driving, which is a prerequisite for risk assessment and effective decision making. Although there exist a lot of works dealing with modeling driver behavior of a single object, it remains a challenge to make predictions for multiple highly interactive agents that react to each other simultaneously. In this work, we propose a generic probabilistic hierarchical recognition and prediction framework which employs a two-layer Hidden Markov Model (TLHMM) to obtain the distribution of potential situations and a learning-based dynamic scene evolution model to sample a group of future trajectories. Instead of predicting motions of a single entity, we propose to get the joint distribution by modeling multiple interactive agents as a whole system. Moreover, due to the decoupling property of the layered structure, our model is suitable for knowledge transfer from simulation to real world applications as well as among different traffic scenarios, which can reduce the computational efforts of training and the demand for a large data amount. A case study of highway ramp merging scenario is demonstrated to verify the effectiveness and accuracy of the proposed framework.

Motivation & Objective

• Address the challenge of predicting joint behaviors in highly interactive traffic scenarios involving multiple agents reacting to each other.
• Overcome the limitations of single-agent prediction models by modeling interactions as a system-level dynamic process.
• Enable knowledge transfer from simulation to real-world applications to reduce reliance on large-scale real-world datasets.
• Develop a robust, uncertainty-aware framework that captures complex human-like interaction patterns in dynamic traffic environments.

Proposed method

• Employ a two-layer Hidden Markov Model (TLHMM) to model high-level situation states and their evolution, with the first layer capturing interaction patterns and the second layer modeling scene dynamics.
• Use a layered HMM architecture to decouple high-level situation recognition from low-level motion prediction, enabling transfer learning across scenarios and domains.
• Apply the Expectation-Maximization (EM) algorithm to train the TLHMM on synthetic data from a driving simulator, followed by fine-tuning on real-world NGSIM data.
• Use a Gaussian Mixture Model (GMM) to generate diverse, probabilistic future trajectories based on the recognized situation state, capturing uncertainty in joint agent motion.
• Incorporate meta-features derived from raw observations to improve robustness to sensor noise and reduce sensitivity to raw feature fluctuations.
• Integrate leading vehicle state (position and velocity) into feature sequences for real-world adaptation, especially in dense traffic conditions.

Experimental results

Research questions

• RQ1 Can a hierarchical HMM framework effectively recognize and predict joint interactive situations among multiple traffic agents in complex scenarios?
• RQ2 How well does the proposed TLHMM model generalize from simulation to real-world driving data in terms of recognition accuracy and computational efficiency?
• RQ3 To what extent does transfer learning reduce the need for large-scale real-world training data while maintaining prediction performance?
• RQ4 How does the model’s robustness to observation noise and feature fluctuations compare to standard HMM and QDA baselines?
• RQ5 Can the framework generate accurate, long-term joint trajectory distributions that align with ground truth in real-world scenarios?

Key findings

• The TLHMM-based model outperformed standard HMM and QDA in recognizing true interaction outcomes earlier and with less fluctuation, demonstrating superior robustness to noise and feature variability.
• The model achieved accurate long-term trajectory predictions, as shown by the ground truth trajectory lying near the mean of the predicted GMM distribution in the occupancy heatmap.
• Fine-tuning the pre-trained second-layer HMM on real-world NGSIM data required only 35 EM iterations on average, compared to 112 iterations when training from scratch, significantly reducing computational cost.
• The transferred model with fine-tuning achieved performance comparable to a model trained from scratch, validating the effectiveness of knowledge transfer from simulation to real-world applications.
• The layered structure enabled effective decoupling of situation recognition and motion prediction, enhancing model interpretability and adaptability across different traffic scenarios.