[Paper Review] Decoupling Localization and Classification in Single Shot Temporal Action Detection

Yupan Huang, Qi Dai+1 more|arXiv (Cornell University)|Apr 16, 2019

Human Pose and Action Recognition21 references4 citations

TL;DR

This paper proposes Decouple-SSAD, a novel one-stage temporal action detection framework that decouples localization and classification into separate branches to learn specialized features, improving performance. By using deconvolution to refine feature maps from deeper layers and training end-to-end, it achieves state-of-the-art mAP@0.5 of 35.8% on THUMOS14, outperforming prior one-stage methods and approaching two-stage SOTA.

ABSTRACT

Video temporal action detection aims to temporally localize and recognize the action in untrimmed videos. Existing one-stage approaches mostly focus on unifying two subtasks, i.e., localization of action proposals and classification of each proposal through a fully shared backbone. However, such design of encapsulating all components of two subtasks in one single network might restrict the training by ignoring the specialized characteristic of each subtask. In this paper, we propose a novel Decoupled Single Shot temporal Action Detection (Decouple-SSAD) method to mitigate such problem by decoupling the localization and classification in a one-stage scheme. Particularly, two separate branches are designed in parallel to enable each component to own representations privately for accurate localization or classification. Each branch produces a set of action anchor layers by applying deconvolution to the feature maps of the main stream. Each branch produces a set of feature maps by applying deconvolution to the feature maps of the main stream. High-level semantic information from deeper layers is thus incorporated to enhance the feature representations. We conduct extensive experiments on THUMOS14 dataset and demonstrate superior performance over state-of-the-art methods. Our code is available online.

Motivation & Objective

• To address the limitation of one-stage temporal action detection methods that jointly optimize localization and classification using a shared backbone, which may hinder specialized feature learning.
• To improve detection accuracy by enabling separate optimization of proposal generation and action classification through dedicated branches.
• To incorporate high-level semantic features from deeper layers into anchor feature maps using deconvolution, enhancing representation quality.
• To achieve state-of-the-art performance on the standard THUMOS14 benchmark while maintaining an end-to-end one-stage training paradigm.

Proposed method

• The framework introduces two parallel branches—classification and proposal—derived from a shared main stream, decoupling the optimization of localization and classification.
• Each branch applies deconvolution to feature maps from deeper layers to upsample and refine the representations, incorporating high-level semantic context.
• Action anchor layers are generated in each branch by applying deconvolution to the main stream’s feature maps, enabling specialized feature learning for each task.
• The entire network is trained end-to-end, allowing joint optimization while preserving task-specific feature representations.
• A refinement branch is evaluated as an ablation, which combines localization and classification in a single branch using deconvolution, but is less effective than the decoupled approach.
• The method uses a multi-branch anchor network architecture that maintains the one-stage inference efficiency while improving accuracy.

Experimental results

Research questions

• RQ1 Can decoupling localization and classification in a one-stage framework improve temporal action detection performance by enabling specialized feature learning?
• RQ2 How does incorporating high-level semantic features from deeper layers via deconvolution affect localization and classification accuracy?
• RQ3 Does a dedicated branch for each task outperform a shared branch that jointly optimizes both tasks in one-stage detection?
• RQ4 What is the contribution of separate branches versus a unified refinement branch in improving mAP on THUMOS14?
• RQ5 Can a decoupled one-stage method achieve performance comparable to state-of-the-art two-stage approaches?

Key findings

• Decouple-SSAD achieves a mAP@0.5 of 35.8% on the THUMOS14 dataset, surpassing the best one-stage baseline (SSAD) by 11.2 percentage points.
• The ablation study shows that the classification branch alone improves mAP@0.5 by 1.0% and the proposal branch by 2.2% over the baseline.
• Combining both branches yields a 4.6% improvement over the baseline, demonstrating the effectiveness of decoupled feature learning.
• The refinement branch, which combines both tasks in one path, only improves performance by 2.8%, confirming that decoupling is more effective than joint refinement.
• Pre-training on Kinetics boosts performance to 44.2% mAP@0.5, indicating strong generalization and scalability of the proposed architecture.
• The method outperforms the best one-stage competitor (SS-TAD) by 6.6% mAP@0.5 and is within 1.1% of the best two-stage method (BSN), demonstrating strong competitiveness.