[Paper Review] Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos

Rui Hou, Chen Chen+1 more|arXiv (Cornell University)|Mar 30, 2017

Human Pose and Action Recognition24 references58 citations

TL;DR

An end-to-end 3D-CNN framework (T-CNN) that detects and localizes actions in videos by generating and linking 3D tube proposals, using Tube Proposal Network and Tube-of-Interest pooling for spatio-temporal action detection.

ABSTRACT

Deep learning has been demonstrated to achieve excellent results for image classification and object detection. However, the impact of deep learning on video analysis (e.g. action detection and recognition) has been limited due to complexity of video data and lack of annotations. Previous convolutional neural networks (CNN) based video action detection approaches usually consist of two major steps: frame-level action proposal detection and association of proposals across frames. Also, these methods employ two-stream CNN framework to handle spatial and temporal feature separately. In this paper, we propose an end-to-end deep network called Tube Convolutional Neural Network (T-CNN) for action detection in videos. The proposed architecture is a unified network that is able to recognize and localize action based on 3D convolution features. A video is first divided into equal length clips and for each clip a set of tube proposals are generated next based on 3D Convolutional Network (ConvNet) features. Finally, the tube proposals of different clips are linked together employing network flow and spatio-temporal action detection is performed using these linked video proposals. Extensive experiments on several video datasets demonstrate the superior performance of T-CNN for classifying and localizing actions in both trimmed and untrimmed videos compared to state-of-the-arts.

Motivation & Objective

• Motivate the need for end-to-end spatio-temporal action detection in videos.
• Propose a unified 3D-CNN framework that directly localizes and recognizes actions in video clips.
• Introduce a Tube Proposal Network (TPN) to generate tube proposals from 3D features.
• Develop Tube-of-Interest (ToI) pooling to produce fixed-length descriptors for variable tube proposals.
• Demonstrate state-of-the-art performance on trimmed and untrimmed video datasets.

Proposed method

• Process video clips with a 3D ConvNet to extract spatio-temporal feature cubes.
• Generate tube proposals per clip using a Tube Proposal Network (TPN) with actionness scoring and anchor boxes learned via k-means.
• Link tube proposals across adjacent clips using an actionness and overlap-based scoring and network flow.
• Apply Tube-of-Interest (ToI) pooling to obtain fixed-length features from linked tube proposals for action classification.
• Train end-to-end with alternating updates between TPN and the recognition network, using 1x1 conv to match dimensions and final fully connected layers for bbox regression and action classification.
• Use temporal skip pooling to preserve frame order information by mapping conv5 proposals to conv2 feature tubes across eight frames per clip.

Experimental results

Research questions

• RQ1 Can an end-to-end 3D CNN framework learn to localize and recognize actions directly from video inputs without relying on two-stream or frame-level proposals?
• RQ2 Does a Tube Proposal Network with data-driven anchor boxes improve spatio-temporal action localization compared to frame-based proposals?
• RQ3 Can ToI pooling effectively produce fixed-length descriptors for variable-length tubes to enable robust action classification?
• RQ4 Does temporal skip pooling preserve temporal order information and improve localization accuracy?
• RQ5 How does T-CNN perform on trimmed and untrimmed videos across multiple datasets?

Key findings

• T-CNN achieves state-of-the-art performance on trimmed datasets UCF-Sports, J-HMDB, and UCF-101 and on the untrimmed THUMOS’14 dataset.
• Using 3D ConvNet-based tube proposals and ToI pooling yields improved action localization and recognition.
• Temporal skip pooling preserves temporal order information, improving localization accuracy.
• An end-to-end approach operating on 3D volumes with learnable anchors (via k-means) outperforms methods relying on frame-level proposals or two-stream architectures.
• The approach demonstrates strong action recognition accuracy: 95.7% on UCF-Sports, 67.2% on J-HMDB, and 94.4% on UCF-101 (24 actions).
• On THUMOS’14, negative mining further boosts performance.