[Paper Review] Cascaded Boundary Regression for Temporal Action Detection

Jiyang Gao, Zhenheng Yang+1 more|arXiv (Cornell University)|May 2, 2017

Human Pose and Action Recognition16 references58 citations

TL;DR

The paper introduces Cascaded Boundary Regression (CBR) within a two-stage temporal action detection pipeline to iteratively refine action boundaries, achieving state-of-the-art results on THUMOS-14 and TVSeries, especially at higher IoU thresholds.

ABSTRACT

Temporal action detection in long videos is an important problem. State-of-the-art methods address this problem by applying action classifiers on sliding windows. Although sliding windows may contain an identifiable portion of the actions, they may not necessarily cover the entire action instance, which would lead to inferior performance. We adapt a two-stage temporal action detection pipeline with Cascaded Boundary Regression (CBR) model. Class-agnostic proposals and specific actions are detected respectively in the first and the second stage. CBR uses temporal coordinate regression to refine the temporal boundaries of the sliding windows. The salient aspect of the refinement process is that, inside each stage, the temporal boundaries are adjusted in a cascaded way by feeding the refined windows back to the system for further boundary refinement. We test CBR on THUMOS-14 and TVSeries, and achieve state-of-the-art performance on both datasets. The performance gain is especially remarkable under high IoU thresholds, e.g. map@tIoU=0.5 on THUMOS-14 is improved from 19.0% to 31.0%.

Motivation & Objective

• Motivate precise temporal localization in untrimmed videos beyond sliding-window coverage.
• Propose a cascaded boundary regression mechanism to progressively refine temporal boundaries within each stage.
• Demonstrate effectiveness of CBR on both temporal action proposal generation and action detection.
• Evaluate performance on THUMOS-14 and TVSeries datasets against prior methods.

Proposed method

• Two-stage action detection pipeline: stage 1 generates class-agnostic temporal proposals; stage 2 performs action-specific detection based on proposals.
• Unit-level video feature extraction using C3D and two-stream CNN features with context-augmented clip representations.
• Temporal coordinate regression using non-parameterized unit-level offsets to refine start/end boundaries.
• Cascaded boundary regression within each stage: iteratively feed refined clips back into the same network for further boundary refinement (K_p steps for proposals, K_d steps for detection).
• Multi-task loss combining classification (binary for proposals, multi-class for detection) with L1-based boundary regression, optimized via Adam with specified hyperparameters.
• Training samples are drawn from sliding windows with tIoU-based labeling, enabling separate training of proposal and detection networks.

Experimental results

Research questions

• RQ1 Does non-parameterized unit-level temporal coordinate regression outperform parameterized and frame-level offsets for boundary refinement?
• RQ2 Do cascaded boundary regression steps improve boundary localization and action detection performance over a single-step regression?
• RQ3 How does CBR perform for both temporal proposal generation and action detection on THUMOS-14 and TVSeries compared to prior methods?
• RQ4 What is the impact of using CBR with different feature types (C3D vs. two-stream) on localization accuracy?

Key findings

• Unit-level, non-parameterized temporal offsets outperform parameterized and frame-level approaches for boundary regression.
• Cascaded boundary regression improves proposal AR@F=1.0 and detection mAP@IoU=0.5 over non-cascaded baselines, with best results at intermediate cascade depths (e.g., K_p=3 for proposals, K_d=2 for detection).
• CBR with two-stream features achieves state-of-the-art AR@F=1.0 and mAP@tIoU=0.5 on THUMOS-14, significantly outperforming prior methods at high IoU thresholds.
• On THUMOS-14, CBR-C3D and CBR-TS surpass SCNN-prop and TURN under various metrics, with CBR-TS achieving 31.0% mAP@tIoU=0.5 for detection.
• On TVSeries, cascaded regression yields substantial gains over non-regression baselines, and CBR-TS outperforms prior FV and SVM-TS approaches in several tIoU settings.
• The results demonstrate strong effectiveness of CBR for both proposal generation and action detection across challenging datasets.