[Paper Review] R-C3D: Region Convolutional 3D Network for Temporal Activity Detection

Huijuan Xu, Abir Das+1 more|arXiv (Cornell University)|Mar 22, 2017

Human Pose and Action RecognitionComputer Science35 references129 citations

TL;DR

R-C3D is an end-to-end, fast temporal activity detection model that shares 3D CNN features between proposal and classification stages, achieving state-of-the-art results on THUMOS’14 and strong generalization to ActivityNet and Charades.

ABSTRACT

We address the problem of activity detection in continuous, untrimmed video streams. This is a difficult task that requires extracting meaningful spatio-temporal features to capture activities, accurately localizing the start and end times of each activity. We introduce a new model, Region Convolutional 3D Network (R-C3D), which encodes the video streams using a three-dimensional fully convolutional network, then generates candidate temporal regions containing activities, and finally classifies selected regions into specific activities. Computation is saved due to the sharing of convolutional features between the proposal and the classification pipelines. The entire model is trained end-to-end with jointly optimized localization and classification losses. R-C3D is faster than existing methods (569 frames per second on a single Titan X Maxwell GPU) and achieves state-of-the-art results on THUMOS'14. We further demonstrate that our model is a general activity detection framework that does not rely on assumptions about particular dataset properties by evaluating our approach on ActivityNet and Charades. Our code is available at http://ai.bu.edu/r-c3d/.

Motivation & Objective

• Define a fast, end-to-end model for detecting and localizing activities in untrimmed videos.
• Learn shared spatio-temporal features for both proposal generation and activity classification.
• Enable variable-length activity proposals using 3D RoI pooling and anchor-based proposals.
• Improve efficiency over sliding-window and regression-only methods by sharing features.
• Demonstrate generality across diverse datasets (THUMOS’14, ActivityNet, Charades).

Proposed method

• Encode video streams with a fully convolutional 3D network (C3D-based) to extract shared spatio-temporal features.
• Propose temporal regions likely to contain activities using anchor-based segments and a temporal proposal subnet.
• Extend 2D RoI pooling to 3D RoI pooling to obtain fixed-size features for variable-length proposals.
• Classify proposals and refine boundaries with a shared classification subtree and regression heads.
• Train end-to-end with joint localization (regression) and classification losses across both subnets.

Experimental results

Research questions

• RQ1 Can an end-to-end 3DConvNet with shared features improve speed and accuracy for temporal activity detection on untrimmed videos?
• RQ2 Do anchor-based temporal proposals plus 3D RoI pooling enable accurate detection of flexible activity boundaries across datasets?
• RQ3 How does R-C3D perform compared to state-of-the-art methods on THUMOS’14, ActivityNet, and Charades?
• RQ4 What is the inference speed benefit of sharing convolutional features between proposal and classification stages?

Key findings

• R-C3D achieves state-of-the-art mAP on THUMOS’14, notably at higher IoU thresholds.
• Two-way buffering (inference on reversed video buffer) improves mAP across IoU thresholds (e.g., from 27.0% to 28.9% at IoU 0.5).
• On THUMOS’14, per-class APs show large gains for activities like Basketball Dunk, Cliff Diving, and Javelin Throw.
• On ActivityNet and Charades, R-C3D demonstrates strong generalization with competitive results using end-to-end training.
• R-C3D runs significantly faster than alternatives (569 fps on Titan X Maxwell; 1030 fps on Titan X Pascal).
• The model is capable of detecting arbitrary-length activities and handling overlapping activities (Charades) via overlapping proposals and boundary refinement.