[Paper Review] A Real-time Vision Framework for Pedestrian Behavior Recognition and Intention Prediction at Intersections Using 3D Pose Estimation

Ue-Hwan Kim, Dongho Ka+2 more|arXiv (Cornell University)|Sep 23, 2020

Autonomous Vehicle Technology and Safety41 references4 citations

TL;DR

This paper proposes a real-time vision framework using 3D pose estimation to recognize pedestrian behaviors (100.53 FPS) and predict crossing intentions (35.76 FPS), achieving 89.3% accuracy on TUD without training and 91.28% on their own dataset, setting a new state of the art through site-independent features and robust generalization.

ABSTRACT

Minimizing traffic accidents between vehicles and pedestrians is one of the primary research goals in intelligent transportation systems. To achieve the goal, pedestrian behavior recognition and prediction of pedestrian's crossing or not-crossing intention play a central role. Contemporary approaches do not guarantee satisfactory performance due to lack of generalization, the requirement of manual data labeling, and high computational complexity. To overcome these limitations, we propose a real-time vision framework for two tasks: pedestrian behavior recognition (100.53 FPS) and intention prediction (35.76 FPS). Our framework obtains satisfying generalization over multiple sites because of the proposed site-independent features. At the center of the feature extraction lies 3D pose estimation. The 3D pose analysis enables robust and accurate recognition of pedestrian behaviors and prediction of intentions over multiple sites. The proposed vision framework realizes 89.3% accuracy in the behavior recognition task on the TUD dataset without any training process and 91.28% accuracy in intention prediction on our dataset achieving new state-of-the-art performance. To contribute to the corresponding research community, we make our source codes public which are available at this https URL

Motivation & Objective

• To address the limitations of existing methods in pedestrian intention prediction and behavior recognition, such as poor generalization, reliance on manual labeling, and high computational cost.
• To develop a real-time system capable of accurately recognizing pedestrian behaviors and predicting crossing intentions at urban intersections.
• To enable site-independent performance through the design of generalizable visual features derived from 3D pose estimation.
• To achieve high accuracy in both behavior recognition and intention prediction without requiring extensive training data or retraining for new locations.

Proposed method

• Leverages 3D pose estimation as the core component for extracting robust, site-independent visual features from monocular RGB video streams.
• Employs a lightweight pipeline to achieve real-time inference at 100.53 FPS for behavior recognition and 35.76 FPS for intention prediction.
• Uses a site-independent feature representation learned from 3D pose sequences to enhance model generalization across diverse intersection environments.
• Applies a classification head on top of 3D pose features to predict pedestrian behaviors (e.g., walking, stopping) and crossing intentions.
• Trains the intention prediction model on a custom dataset, achieving high accuracy without domain-specific fine-tuning.
• Makes the source code publicly available to support reproducibility and community advancement in intelligent transportation systems.

Experimental results

Research questions

• RQ1 Can a vision-based framework using 3D pose estimation achieve real-time performance for pedestrian behavior recognition and intention prediction in urban intersections?
• RQ2 To what extent can site-independent features improve generalization across different intersection environments without retraining?
• RQ3 What level of accuracy can be achieved in pedestrian intention prediction using 3D pose-based features compared to prior methods?
• RQ4 How does the proposed framework perform on benchmark datasets without any training, and what does this imply about feature robustness?
• RQ5 Can the framework maintain high performance across diverse real-world sites with minimal adaptation?

Key findings

• The framework achieves 89.3% accuracy in pedestrian behavior recognition on the TUD dataset without any training, demonstrating strong generalization from learned 3D pose features.
• On their own custom dataset, the framework reaches 91.28% accuracy in intention prediction, setting a new state-of-the-art performance.
• The system operates in real time, processing at 100.53 FPS for behavior recognition and 35.76 FPS for intention prediction, suitable for deployment in dynamic traffic environments.
• The use of 3D pose estimation enables robust feature extraction that generalizes well across multiple intersection sites without retraining.
• The proposed site-independent features significantly reduce the need for data re-collection and manual labeling in new environments.
• The public release of source code supports reproducibility and accelerates future research in vision-based pedestrian behavior understanding.