[Paper Review] ActionXPose: A Novel 2D Multi-view Pose-based Algorithm for Real-time Human Action Recognition

Federico Angelini, Zeyu Fu+3 more|arXiv (Cornell University)|Oct 29, 2018

Human Pose and Action RecognitionComputer Science39 references19 citations

TL;DR

ActionXPose is a real-time, 2D multi-view pose-based human action recognition algorithm that leverages OpenPose-detected body landmarks and combines 1D CNN and LSTM networks for classification. It achieves state-of-the-art performance on posture-level HAR datasets like i3DPost and KTH, with strong generalization across multiple datasets and robustness to viewpoint, camera motion, and subject proximity changes.

ABSTRACT

We present ActionXPose, a novel 2D pose-based algorithm for posture-level Human Action Recognition (HAR). The proposed approach exploits 2D human poses provided by OpenPose detector from RGB videos. ActionXPose aims to process poses data to be provided to a Long Short-Term Memory Neural Network and to a 1D Convolutional Neural Network, which solve the classification problem. ActionXPose is one of the first algorithms that exploits 2D human poses for HAR. The algorithm has real-time performance and it is robust to camera movings, subject proximity changes, viewpoint changes, subject appearance changes and provide high generalization degree. In fact, extensive simulations show that ActionXPose can be successfully trained using different datasets at once. State-of-the-art performance on popular datasets for posture-related HAR problems (i3DPost, KTH) are provided and results are compared with those obtained by other methods, including the selected ActionXPose baseline. Moreover, we also proposed two novel datasets called MPOSE and ISLD recorded in our Intelligent Sensing Lab, to show ActionXPose generalization performance.

Motivation & Objective

• To develop a novel 2D multi-view pose-based algorithm for real-time human action recognition using only RGB video input.
• To improve generalization across diverse datasets by enabling joint training on multiple posture-level datasets.
• To achieve robustness against viewpoint changes, camera movements, subject proximity variations, and appearance changes.
• To introduce two new datasets, MPOSE and ISLD, to evaluate generalization performance on unconstrained recording conditions.
• To demonstrate state-of-the-art performance on standard posture-level HAR benchmarks using minimal input data (14 body landmarks)

Proposed method

• Uses OpenPose to extract 2D body joint coordinates (14 landmarks) from RGB video frames as input features.
• Applies a pre-processing pipeline to normalize pose sequences, compensating for camera motion, zoom, and subject proximity variations.
• Processes the normalized pose sequences through a 1D Convolutional Neural Network (1D-CNN) to extract spatial-temporal features.
• Combines 1D-CNN features with a Long Short-Term Memory (LSTM) network to model long-range temporal dependencies in action sequences.
• Employs an end-to-end training strategy allowing joint learning across multiple datasets, including MPOSE and ISLD.
• Leverages the LSTM’s ability to handle variable-length input sequences, enabling flexibility in temporal modeling without fixed-length constraints.

Experimental results

Research questions

• RQ1 Can a pose-based approach achieve state-of-the-art performance in posture-level human action recognition using only 2D body landmarks from RGB videos?
• RQ2 To what extent can ActionXPose generalize across multiple datasets with varying viewpoints, recording conditions, and action styles?
• RQ3 How effective is the proposed pre-processing in mitigating the effects of camera motion, zoom, and subject proximity changes?
• RQ4 Can joint training on multiple datasets improve model generalization and performance on unseen data?
• RQ5 How does ActionXPose perform in comparison to baseline methods and existing state-of-the-art approaches on standard and newly introduced datasets?

Key findings

• ActionXPose achieves state-of-the-art performance on the i3DPost and KTH datasets, outperforming the OpenPose-baseline and other existing methods.
• The model generalizes effectively across multiple datasets, including MPOSE and ISLD, demonstrating high robustness to viewpoint, camera motion, and subject proximity variations.
• Even with only 14 body landmark coordinates as input, ActionXPose achieves high accuracy, indicating effective feature learning from minimal pose data.
• The model processes sequences of variable length (15 to 358 time-steps in MPOSE), confirming its flexibility and suitability for real-time and online action detection.
• Transfer learning from MPOSE to ISLD shows performance degradation due to differences in action execution styles, suggesting room for improvement in domain generalization.
• ActionXPose runs in real-time, with performance primarily limited by the OpenPose pose detector, confirming its feasibility for real-world surveillance and interactive applications.