[Paper Review] Segmental Spatio-Temporal CNNs for Fine-grained Action Segmentation and Classification

Colin Lea, Austin Reiter+2 more|arXiv (Cornell University)|Feb 9, 2016

Human Pose and Action Recognition3 citations

TL;DR

This paper proposes a segmental spatio-temporal CNN model that integrates spatial CNNs for scene objects and relationships, temporal CNNs for dynamic changes over time, and a semi-Markov model for action transition modeling. The method enables fast, joint action segmentation and classification, achieving state-of-the-art performance on cooking and surgical action datasets with significantly improved inference speed.

ABSTRACT

Joint segmentation and classification of fine-grained actions is important for applications in human-robot interaction, video surveillance, and human skill evaluation. However, despite substantial recent progress in large scale action classification, the performance of state-of-the-art fine-grained action recognition approaches remains low. In this paper, we propose a new spatio-temporal CNN model for fine-grained action classification and segmentation, which combines (1) a spatial CNN to represent objects in the scene and their spatial relationships; (2) a temporal CNN that captures how object relationships within an action change over time; and (3) a semi-Markov model that captures transitions from one action to another. In addition, we introduce an efficient segmental inference algorithm for joint segmentation and classification of actions that is orders of magnitude faster than state-of-the-art approaches. We highlight the effectiveness of our approach on cooking and surgical action datasets for which we observe substantially improved performance relative to recent baseline methods.

Motivation & Objective

• Address the challenge of low performance in fine-grained action recognition despite advances in large-scale action classification.
• Enable joint segmentation and classification of fine-grained actions for applications in human-robot interaction and video surveillance.
• Develop an efficient inference algorithm that drastically reduces computation time compared to existing methods.
• Improve modeling of spatial relationships and temporal dynamics in actions to enhance recognition accuracy.
• Integrate a semi-Markov model to better capture transitions between distinct actions in a video sequence.

Proposed method

• Use a spatial CNN to extract features representing objects and their spatial configurations within each frame.
• Apply a temporal CNN to model how object relationships evolve over time, capturing motion and interaction dynamics.
• Combine spatial and temporal features through a shared representation for end-to-end learning.
• Employ a semi-Markov model to model action boundaries and transitions, enabling segment-level prediction.
• Design a novel segmental inference algorithm that accelerates joint segmentation and classification by orders of magnitude.
• Train the model end-to-end using weakly supervised learning with video-level labels.

Experimental results

Research questions

• RQ1 Can a unified spatio-temporal CNN architecture improve fine-grained action recognition by modeling both spatial relationships and temporal dynamics?
• RQ2 How effective is a semi-Markov model in capturing action transitions compared to standard CRF or RNN-based approaches?
• RQ3 Can a new segmental inference algorithm achieve significantly faster inference without sacrificing accuracy?
• RQ4 Does the proposed method outperform state-of-the-art approaches on benchmark datasets for cooking and surgical actions?
• RQ5 To what extent do spatial and temporal feature representations contribute to improved segmentation and classification performance?

Key findings

• The proposed method achieves substantially improved performance on cooking and surgical action datasets compared to recent baseline methods.
• The segmental inference algorithm is orders of magnitude faster than state-of-the-art approaches, enabling real-time or near-real-time inference.
• The integration of spatial, temporal, and semi-Markov modeling leads to more accurate action boundary detection and classification.
• The model demonstrates strong generalization on fine-grained action recognition tasks where prior methods underperform.
• The ablation study confirms that both spatial and temporal modeling components contribute significantly to the overall performance gain.