[Paper Review] Fast Learning of Temporal Action Proposal via Dense Boundary Generator

Chuming Lin, Jian Li+8 more|arXiv (Cornell University)|Nov 11, 2019

Human Pose and Action Recognition25 references15 citations

TL;DR

This paper proposes Dense Boundary Generator (DBG), a unified framework for fast and accurate temporal action proposal generation in untrimmed videos. By leveraging dual-stream features for boundary classification and action-aware completeness regression via a novel proposal feature generation layer, DBG achieves state-of-the-art performance on ActivityNet-1.3 and THUMOS14, outperforming methods like BMN and MGG with significant mAP gains in action detection.

ABSTRACT

Generating temporal action proposals remains a very challenging problem, where the main issue lies in predicting precise temporal proposal boundaries and reliable action confidence in long and untrimmed real-world videos. In this paper, we propose an efficient and unified framework to generate temporal action proposals named Dense Boundary Generator (DBG), which draws inspiration from boundary-sensitive methods and implements boundary classification and action completeness regression for densely distributed proposals. In particular, the DBG consists of two modules: Temporal boundary classification (TBC) and Action-aware completeness regression (ACR). The TBC aims to provide two temporal boundary confidence maps by low-level two-stream features, while the ACR is designed to generate an action completeness score map by high-level action-aware features. Moreover, we introduce a dual stream BaseNet (DSB) to encode RGB and optical flow information, which helps to capture discriminative boundary and actionness features. Extensive experiments on popular benchmarks ActivityNet-1.3 and THUMOS14 demonstrate the superiority of DBG over the state-of-the-art proposal generator (e.g., MGG and BMN). Our code will be made available upon publication.

Motivation & Objective

• Address the challenge of generating precise temporal action proposals in long, untrimmed videos with variable durations and complex backgrounds.
• Overcome limitations of anchor-based methods (inflexible boundaries) and boundary-based methods (lack of global context) in proposal generation.
• Improve proposal quality by integrating global proposal features for boundary prediction and action completeness estimation.
• Enable end-to-end learning with auxiliary supervision via actionness classification to enhance feature discriminability for completeness regression.

Proposed method

• Propose a Dual Stream BaseNet (DSB) that encodes RGB and optical flow features to extract both low-level boundary-sensitive and high-level action-aware representations.
• Design a Proposal Feature Generation (PFG) layer to transform sequence features into matrix-like representations, capturing global context for each proposal.
• Implement a Temporal Boundary Classification (TBC) module that predicts two confidence maps (start and end boundaries) using low-level dual-stream features.
• Introduce an Action-aware Completeness Regression (ACR) module that generates a completeness score map using high-level actionness features to assess proposal quality.
• Fuse three score maps (start confidence, end confidence, completeness score) through post-processing with Soft-NMS to generate final proposals.
• Train the network with multi-task supervision: actionness classification loss for DSB and regression loss for ACR, with TBC trained via binary cross-entropy.

Experimental results

Research questions

• RQ1 Can a unified framework jointly predict precise temporal boundaries and reliable action completeness scores for dense proposals in untrimmed videos?
• RQ2 How does incorporating global proposal features through a PFG layer improve boundary localization and completeness estimation compared to local feature methods?
• RQ3 To what extent does auxiliary supervision via actionness classification enhance the performance of action completeness regression?
• RQ4 How does the proposed DBG compare to state-of-the-art methods like BMN and MGG in terms of proposal quality and detection accuracy across benchmarks?

Key findings

• DBG achieves a mean Average Precision (mAP) of 45.9% on THUMOS14 at IoU threshold 0.3 using the SCNN classifier, outperforming BMN (45.7%) and MGG (44.9%) by 0.2–1.4 percentage points.
• With the UntrimmedNet classifier, DBG achieves 57.8% mAP at IoU 0.3, surpassing BMN (56.0%) and MGG (53.9%) by 1.8 and 3.9 percentage points, respectively.
• On ActivityNet-1.3, DBG achieves 66.57% AUC on the unseen subset, demonstrating strong generalization to novel action categories.
• Ablation studies confirm that removing the TBC module (which predicts dense boundary confidence maps) leads to a significant AUC drop, especially at high IoU thresholds (e.g., 0.9), proving its importance for boundary precision.
• The ACR module with auxiliary actionness supervision improves completeness estimation, as shown by a 1.5% AUC gain over using only dual-stream features.
• The PFG layer with 8/16/8 sampling from left, center, and right regions of proposals achieves optimal performance, confirming the necessity of global context for accurate proposal generation.