[Paper Review] CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos

Zheng Shou, Jonathan Chan+3 more|arXiv (Cornell University)|Mar 4, 2017

Human Pose and Action Recognition60 references62 citations

TL;DR

Introduces a Convolutional-De-Convolutional (CDC) network built on top of 3D ConvNets to predict frame-level action scores, enabling precise temporal localization in untrimmed videos with high efficiency (≈500 FPS).

ABSTRACT

Temporal action localization is an important yet challenging problem. Given a long, untrimmed video consisting of multiple action instances and complex background contents, we need not only to recognize their action categories, but also to localize the start time and end time of each instance. Many state-of-the-art systems use segment-level classifiers to select and rank proposal segments of pre-determined boundaries. However, a desirable model should move beyond segment-level and make dense predictions at a fine granularity in time to determine precise temporal boundaries. To this end, we design a novel Convolutional-De-Convolutional (CDC) network that places CDC filters on top of 3D ConvNets, which have been shown to be effective for abstracting action semantics but reduce the temporal length of the input data. The proposed CDC filter performs the required temporal upsampling and spatial downsampling operations simultaneously to predict actions at the frame-level granularity. It is unique in jointly modeling action semantics in space-time and fine-grained temporal dynamics. We train the CDC network in an end-to-end manner efficiently. Our model not only achieves superior performance in detecting actions in every frame, but also significantly boosts the precision of localizing temporal boundaries. Finally, the CDC network demonstrates a very high efficiency with the ability to process 500 frames per second on a single GPU server. We will update the camera-ready version and publish the source codes online soon.

Motivation & Objective

• Motivate the need for fine-grained, frame-level temporal localization beyond pre-defined segment proposals.
• Propose a jointly learned CDC filter that down-samples in space while up-sampling in time to preserve frame-level resolution.
• Design an end-to-end CDC network atop 3D ConvNets to produce dense per-frame action scores.
• Demonstrate improved per-frame labeling accuracy and superior temporal localization precision on THUMOS’14 and ActivityNet 2016.

Proposed method

• Replace/augment C3D with CDC filters that perform spatial downsampling (4x4) and temporal upsampling (2x) jointly.
• Adapt FC6/FC7 into CDC6/CDC7 to enable multi-frame outputs and frame-level predictions.
• Attach a frame-wise softmax classifier (CDC8) and train with a frame-level cross-entropy loss.
• Train end-to-end on video windows (32 frames) with SGD, using pre-trained C3D initializations for stability.
• During testing, generate per-frame scores on proposal windows and refine segment boundaries using Gaussian KDE of frame confidences.

Experimental results

Research questions

• RQ1 Can a joint Convolutional-De-Convolutional (CDC) filter simultaneously downsample spatially and upsample temporally to yield frame-level action predictions?
• RQ2 Do frame-level predictions significantly improve temporal boundary localization compared to segment-level approaches?
• RQ3 How does end-to-end CDC-based localization compare to state-of-the-art methods on THUMOS’14 and ActivityNet 2016?
• RQ4 Is the CDC approach computationally efficient enough for real-time or near-real-time processing?

Key findings

• CDC achieves state-of-the-art per-frame labeling mAP on THUMOS’14, outperforming single-frame, two-stream, LSTM, and earlier C3D-based methods.
• CDC with frame-level predictions yields superior temporal localization accuracy across IoU thresholds (0.3–0.7) compared to S-CNN, C3D+LinearInterp, Conv&De-conv baselines, and CDC variants.
• Refining segment boundaries with CDC frame-level predictions improves temporal localization mAP on ActivityNet 2016, notably at higher IoU (0.75).
• CDC network processes roughly 500 frames per second on a single GPU (Titan X) and requires ~1 GB storage, enabling efficient dense prediction over untrimmed videos.
• End-to-end training and fine-tuning of CDC layers on top of 3D ConvNets provide better discrimination for temporal dynamics than freezing 3D ConvNet features.
• Fine-grained per-frame predictions enable precise boundary refinement even when starting from coarse segment proposals.