[Paper Review] STEP: Spatio-Temporal Progressive Learning for Video Action Detection

Xitong Yang, Xiaodong Yang+4 more|arXiv (Cornell University)|Apr 19, 2019

Human Pose and Action Recognition35 references4 citations

TL;DR

This paper proposes STEP, a spatio-temporal progressive learning framework for video action detection that iteratively refines a small number of initial proposals through spatial refinement and adaptive temporal extension. By progressively improving localization and incorporating longer-range context, STEP achieves state-of-the-art mAP of 75.0% on UCF101 and 18.6% on AVA using only 11 and 34 initial proposals, respectively, while running at 21 fps on a single GPU.

ABSTRACT

In this paper, we propose Spatio-TEmporal Progressive (STEP) action detector---a progressive learning framework for spatio-temporal action detection in videos. Starting from a handful of coarse-scale proposal cuboids, our approach progressively refines the proposals towards actions over a few steps. In this way, high-quality proposals (i.e., adhere to action movements) can be gradually obtained at later steps by leveraging the regression outputs from previous steps. At each step, we adaptively extend the proposals in time to incorporate more related temporal context. Compared to the prior work that performs action detection in one run, our progressive learning framework is able to naturally handle the spatial displacement within action tubes and therefore provides a more effective way for spatio-temporal modeling. We extensively evaluate our approach on UCF101 and AVA, and demonstrate superior detection results. Remarkably, we achieve mAP of 75.0% and 18.6% on the two datasets with 3 progressive steps and using respectively only 11 and 34 initial proposals.

Motivation & Objective

• To address the challenge of spatial displacement in action tubes during spatio-temporal action detection.
• To improve temporal modeling efficiency and accuracy by adaptively extending proposal sequences over time.
• To reduce computational cost by using only a handful of initial proposals instead of thousands of dense anchors.
• To develop an end-to-end progressive optimization framework that outperforms one-shot detection methods.
• To enable robust detection across diverse action scales and motion patterns through iterative refinement.

Proposed method

• The framework uses a multi-step optimization process starting from coarse-scale proposal cuboids.
• Spatial refinement iteratively updates bounding boxes using regression outputs from previous steps to better track moving actors.
• Temporal extension adaptively increases the length of input sequences at each step to incorporate longer-range context.
• The method leverages early fusion of RGB and optical flow features for improved spatio-temporal representation.
• Proposals are refined sequentially, with each step using the output of the prior as input, enabling progressive localization accuracy.
• The framework is trained end-to-end and optimized via a progressive learning strategy that balances accuracy and efficiency.

Experimental results

Research questions

• RQ1 Can a progressive learning framework improve spatio-temporal action detection by iteratively refining proposals?
• RQ2 How does adaptive temporal extension enhance classification accuracy while handling spatial displacement?
• RQ3 Can high detection performance be achieved with only a small number of initial proposals?
• RQ4 Does progressive refinement outperform one-shot detection in terms of accuracy and efficiency?
• RQ5 How does the framework handle action tubes with significant spatial displacement over time?

Key findings

• STEP achieves a frame-mAP of 75.0% on UCF101, representing a 5.5% improvement over the second-best method.
• On AVA, STEP achieves a frame-mAP of 18.6%, outperforming the previous state-of-the-art ACRN by 1.2%.
• The method runs at 21 fps using only 11 initial proposals and 3 steps, outperforming frame-based methods (4 fps) and matching clip-based baselines (23 fps).
• The progressive refinement process significantly reduces misclassification, as shown by the elimination of blue (incorrect) boxes in qualitative results over steps.
• Spatial refinement effectively moves proposals toward actors, improving localization accuracy across diverse action scales, as demonstrated in Figure 8.
• Increasing the number of initial proposals improves accuracy but slows inference, showing a controllable trade-off between speed and performance.