[Paper Review] Scale Matters: Temporal Scale Aggregation Network for Precise Action Localization in Untrimmed Videos

Guoqiang Gong, Liangfeng Zheng+2 more|arXiv (Cornell University)|Aug 2, 2019

Human Pose and Action Recognition57 references4 citations

TL;DR

This paper proposes TSA-Net, a temporal scale aggregation network that uses multi-dilation temporal convolution (MDC) blocks in a multi-branch architecture to handle variable-duration action instances in untrimmed videos. By detecting three critical boundary points (start, mid, end) and fusing multi-scale contextual features, TSA-Net achieves state-of-the-art performance with 46.9% mAP@0.5 on THUMOS14 and 34.62% average mAP on ActivityNet-1.3.

ABSTRACT

Temporal action localization is a recently-emerging task, aiming to localize video segments from untrimmed videos that contain specific actions. Despite the remarkable recent progress, most two-stage action localization methods still suffer from imprecise temporal boundaries of action proposals. This work proposes a novel integrated temporal scale aggregation network (TSA-Net). Our main insight is that ensembling convolution filters with different dilation rates can effectively enlarge the receptive field with low computational cost, which inspires us to devise multi-dilation temporal convolution (MDC) block. Furthermore, to tackle video action instances with different durations, TSA-Net consists of multiple branches of sub-networks. Each of them adopts stacked MDC blocks with different dilation parameters, accomplishing a temporal receptive field specially optimized for specific-duration actions. We follow the formulation of boundary point detection, novelly detecting three kinds of critical points (ie, starting / mid-point / ending) and pairing them for proposal generation. Comprehensive evaluations are conducted on two challenging video benchmarks, THUMOS14 and ActivityNet-1.3. Our proposed TSA-Net demonstrates clear and consistent better performances and re-calibrates new state-of-the-art on both benchmarks. For example, our new record on THUMOS14 is 46.9% while the previous best is 42.8% under mAP@0.5.

Motivation & Objective

• Address the challenge of imprecise temporal boundary localization in untrimmed videos due to variable action durations.
• Overcome the limitations of fixed-receptive-field models and single-scale approaches in handling diverse action lengths.
• Improve proposal quality by fusing multi-scale temporal context through a novel multi-branch network design.
• Enhance boundary detection reliability by jointly regressing start, mid, and end points with a learnable compatibility function.
• Achieve state-of-the-art performance on large-scale benchmarks without relying on heavy spatio-temporal features.

Proposed method

• Propose a multi-dilation temporal convolution (MDC) block that uses dilated convolutions with varying rates to expand the receptive field efficiently.
• Design a three-branch network architecture, each specialized for different action duration scales via distinct dilation rates in stacked MDC blocks.
• Formulate action localization as a boundary point detection task, detecting three types of critical points: start, mid, and end.
• Generate proposals by pairing detected start and end points, with confidence scores computed using a learned pairwise compatibility function.
• Integrate a lightweight sub-network to regress Bayesian confidence for each proposal, improving localization accuracy.
• Use a two-stage framework: first generate proposals via boundary detection, then refine them using the compatibility-aware scoring mechanism.

Experimental results

Research questions

• RQ1 Can multi-dilation temporal convolutions effectively capture long-range temporal context across diverse action durations?
• RQ2 Does a multi-branch architecture with specialized dilation rates improve boundary localization compared to single-branch models?
• RQ3 Can joint detection of start, mid, and end points enhance proposal quality compared to traditional two-point detection?
• RQ4 How effective is the proposed compatibility function in improving proposal scoring and reducing false positives?
• RQ5 To what extent does the proposed method outperform existing state-of-the-art methods on standard benchmarks like THUMOS14 and ActivityNet-1.3?

Key findings

• TSA-Net achieves 46.9% mAP@0.5 on THUMOS14, significantly outperforming the previous SOTA of 42.8% under the same IoU threshold.
• On ActivityNet-1.3, TSA-Net achieves 34.62% average mAP, setting a new state-of-the-art compared to the prior best of 32.84%.
• The proposed compatibility function improves AR@AN by 2.0 points on THUMOS14, demonstrating its effectiveness in refining proposal scores.
• Using P3D features, TSA-Net achieves 48.3% mAP@0.3 on THUMOS14, showing strong generalization across IoU thresholds.
• The multi-branch design with different dilation rates consistently improves performance across all IoU thresholds from 0.3 to 0.7.
• Ablation studies confirm that both the MDC block and the three-point detection strategy are essential for the model’s superior performance.