[Paper Review] P-CNN: Pose-based CNN Features for Action Recognition

Guilhem Chéron, Ivan Laptev+1 more|arXiv (Cornell University)|Jun 11, 2015

Human Pose and Action Recognition33 references4 citations

TL;DR

This paper proposes P-CNN, a pose-based convolutional neural network descriptor that extracts appearance and motion features from tracked human body parts in videos, aggregating them over time using CNNs. It achieves state-of-the-art performance on JHMDB and MPII Cooking datasets, especially excelling in fine-grained action recognition and showing strong complementarity with dense trajectory features.

ABSTRACT

This work targets human action recognition in video. While recent methods typically represent actions by statistics of local video features, here we argue for the importance of a representation derived from human pose. To this end we propose a new Pose-based Convolutional Neural Network descriptor (P-CNN) for action recognition. The descriptor aggregates motion and appearance information along tracks of human body parts. We investigate different schemes of temporal aggregation and experiment with P-CNN features obtained both for automatically estimated and manually annotated human poses. We evaluate our method on the recent and challenging JHMDB and MPII Cooking datasets. For both datasets our method shows consistent improvement over the state of the art.

Motivation & Objective

• To improve fine-grained human action recognition by leveraging structured spatial-temporal information from human pose.
• To design a robust CNN-based descriptor that captures motion and appearance at body part level, improving over global video descriptors.
• To evaluate the effectiveness of P-CNN features under both manually annotated and automatically estimated human poses.
• To demonstrate the complementarity of P-CNN with existing methods like IDT-FV, enhancing overall recognition accuracy.
• To validate that pose-based representations are crucial for capturing discriminative motion patterns in subtle actions.

Proposed method

• P-CNN extracts appearance (RGB) and motion (optical flow) features from localized patches around tracked human body parts across video frames.
• For each body part and frame, a pre-trained CNN is used to extract deep features, preserving spatial and temporal structure.
• Temporal aggregation is performed using min and max pooling over time to form static and dynamic descriptors per body part.
• Descriptors from all body parts are normalized and concatenated to form final appearance and optical flow features.
• The final P-CNN feature vector is the concatenation of appearance and flow descriptors across all body parts and aggregation schemes.
• The method is evaluated using both manually annotated (GT) and automatically estimated human poses to assess robustness.

Experimental results

Research questions

• RQ1 Can pose-based CNN features improve action recognition performance, especially for fine-grained actions, compared to global video descriptors?
• RQ2 How does P-CNN perform when using automatically estimated human poses versus manually annotated ones?
• RQ3 To what extent do P-CNN features complement existing methods like IDT-FV in action recognition?
• RQ4 Does temporal aggregation of CNN features at the body part level enhance discriminative power for subtle actions?
• RQ5 Is the proposed P-CNN descriptor more robust to pose estimation errors than prior pose-based methods like HLPF?

Key findings

• On the JHMDB-GT dataset, P-CNN achieves a mean accuracy of 79.5% when combined with IDT-FV, outperforming the previous state of the art.
• On the MPII Cooking dataset, P-CNN combined with IDT-FV reaches 71.4% mAP, surpassing the previous SOTA of 70.5% and matching the best reported result.
• P-CNN significantly outperforms HLPF on the MPII Cooking dataset, especially with automatically estimated poses, demonstrating robustness to pose estimation errors.
• For actions like 'shoot_gun', 'wave', and 'throw', P-CNN shows large improvements over IDT-FV, with ranking gains from over 200 to under 30 in some cases.
• The method shows a performance drop on 'kick_ball' compared to IDT-FV, indicating that highly dynamic actions are better captured by global motion features.
• Qualitative analysis confirms that P-CNN excels in recognizing localized, fine-grained actions due to its focus on body part-level motion and appearance.