[Paper Review] When Vehicles See Pedestrians with Phones:A Multi-Cue Framework for Recognizing Phone-based Activities of Pedestrians

Akshay Rangesh, Mohan M. Trivedi|arXiv (Cornell University)|Jan 24, 2018

Human-Automation Interaction and Safety4 citations

TL;DR

This paper proposes a vision-based multi-cue framework for recognizing phone-related pedestrian activities—texting, phone calls, or no distraction—using articulated pose estimation, gaze tracking, and exemplar SVMs for object label transfer. The method integrates a Gaussian Process Dynamical Model (GPDM)-based particle filter for efficient, accurate pose tracking across video sequences, achieving high accuracy with low computational cost on a novel annotated dataset.

ABSTRACT

The intelligent vehicle community has devoted considerable efforts to model driver behavior, and in particular to detect and overcome driver distraction in an effort to reduce accidents caused by driver negligence. However, as the domain increasingly shifts towards autonomous and semi-autonomous solutions, the driver is no longer integral to the decision making process, indicating a need to refocus efforts elsewhere. To this end, we propose to study pedestrian distraction instead. In particular, we focus on detecting pedestrians who are engaged in secondary activities involving their cellphones and similar handheld multimedia devices from a purely vision-based standpoint. To achieve this objective, we propose a pipeline incorporating articulated human pose estimation, followed by a soft object label transfer from an ensemble of exemplar SVMs trained on the nearest neighbors in pose feature space. We additionally incorporate head gaze features and prior pose information to carry out cellphone related pedestrian activity recognition. Finally, we offer a method to reliably track the articulated pose of a pedestrian through a sequence of images using a particle filter with a Gaussian Process Dynamical Model (GPDM), which can then be used to estimate sequentially varying activity scores at a very low computational cost. The entire framework is fast (especially for sequential data) and accurate, and easily extensible to include other secondary activities and sources of distraction.

Motivation & Objective

• To address the rising incidence of pedestrian fatalities linked to cellphone use, especially in the context of increasingly autonomous vehicles.
• To develop a real-time, vision-based system for classifying pedestrian distraction into three categories: texting, phone calls, or no secondary activity.
• To improve accuracy and efficiency in activity recognition by leveraging multi-modal cues including pose, gaze, and hand-object interactions.
• To enable robust, low-cost sequential tracking of pedestrian poses using a GPDM-based particle filter with sparse pose measurements.

Proposed method

• A deep convolutional neural network (ConvNet) is used to estimate articulated human pose from single images, providing keypoint coordinates for the head, neck, shoulders, elbows, and wrists.
• Exemplar Support Vector Machines (ESVMs) are trained offline on nearest neighbors in pose feature space to transfer hand-object labels (e.g., phone presence) to test pedestrians.
• Head gaze direction is estimated using a pre-trained ConvNet, providing an additional cue for detecting phone use.
• A Gaussian Process Dynamical Model (GPDM) is integrated into a particle filter framework to model temporal dynamics and improve pose tracking robustness.
• Activity scores from pose, gaze, and object cues are fused using a late-fusion SVM to produce final classification decisions.
• The system operates efficiently by updating the ensemble of ESVMs only every 50 frames, reducing computational load without sacrificing accuracy.

Experimental results

Research questions

• RQ1 Can a vision-based system reliably detect and classify pedestrian distraction into texting, phone calls, or no activity using multi-cue features?
• RQ2 How effective is the integration of pose, gaze, and object interaction cues in improving classification accuracy compared to single-cue approaches?
• RQ3 To what extent can a GPDM-based particle filter enhance pose tracking accuracy and reduce computational cost in sequential pedestrian video analysis?
• RQ4 How does the proposed framework perform under dynamic changes in pedestrian behavior and viewing angles in real-world sequences?

Key findings

• The proposed framework achieves high accuracy in classifying pedestrian activities, with predicted labels closely matching ground truth across all four test sequences, even during rapid activity transitions.
• The GPDM-based particle filter maintains reliable pose tracking with measurement updates as infrequent as once every 5 frames (5 Hz), achieving PCK scores of 0.980 for head and 0.975 for neck.
• Tracking with measurements at 5 Hz yields results comparable to full-frame tracking, demonstrating that sparse pose estimation is sufficient for robust performance.
• The system reduces computational cost by updating the ESVM ensemble every 50 frames, while maintaining high classification accuracy.
• The framework shows strong generalization and scalability, with performance expected to improve as more diverse training data is added.
• The method effectively fuses multiple cues—pose, gaze, and object interaction—resulting in consistent and accurate activity predictions across complex real-world scenarios.