[Paper Review] Bottom-Up Temporal Action Localization with Mutual Regularization

Peisen Zhao, Lingxi Xie+4 more|arXiv (Cornell University)|Feb 18, 2020

Human Pose and Action Recognition41 references8 citations

TL;DR

This paper proposes a bottom-up temporal action localization framework that improves frame-level action phase prediction through mutual regularization, introducing Intra-phase Consistency (IntraC) and Inter-phase Consistency (InterC) losses to enforce temporal smoothness within each phase and ordered consistency across starting, continuing, and ending phases. The method achieves a 6.8% absolute mAP gain at IoU=0.7 on THUMOS14 and generalizes across different TAL architectures.

ABSTRACT

Recently, temporal action localization (TAL), i.e., finding specific action segments in untrimmed videos, has attracted increasing attentions of the computer vision community. State-of-the-art solutions for TAL involves evaluating the frame-level probabilities of three action-indicating phases, i.e. starting, continuing, and ending; and then post-processing these predictions for the final localization. This paper delves deep into this mechanism, and argues that existing methods, by modeling these phases as individual classification tasks, ignored the potential temporal constraints between them. This can lead to incorrect and/or inconsistent predictions when some frames of the video input lack sufficient discriminative information. To alleviate this problem, we introduce two regularization terms to mutually regularize the learning procedure: the Intra-phase Consistency (IntraC) regularization is proposed to make the predictions verified inside each phase; and the Inter-phase Consistency (InterC) regularization is proposed to keep consistency between these phases. Jointly optimizing these two terms, the entire framework is aware of these potential constraints during an end-to-end optimization process. Experiments are performed on two popular TAL datasets, THUMOS14 and ActivityNet1.3. Our approach clearly outperforms the baseline both quantitatively and qualitatively. The proposed regularization also generalizes to other TAL methods (e.g., TSA-Net and PGCN). code: https://github.com/PeisenZhao/Bottom-Up-TAL-with-MR

Motivation & Objective

• To address inconsistent and discontinuous frame-level predictions in bottom-up temporal action localization due to isolated modeling of starting, continuing, and ending phases.
• To exploit temporal dependencies between frames and phase ordering constraints that are ignored in standard binary classification setups.
• To improve localization accuracy by enforcing self-consistency in predictions through end-to-end differentiable regularization.
• To demonstrate generalization of the proposed regularization beyond the base model, enhancing performance in other SOTA TAL frameworks.

Proposed method

• Introduces Intra-phase Consistency (IntraC) regularization to enforce smoothness within positive and negative regions of each phase (start, continue, end), minimizing intra-region discrepancies.
• Proposes Inter-phase Consistency (InterC) regularization to enforce the logical ordering of action phases by coupling predictions between continuing-starting and continuing-ending probabilities.
• Couples the three phase classifiers via consistency checks, enabling mutual regularization during end-to-end training without breaking gradient flow.
• Uses a differentiable loss function combining standard cross-entropy with IntraC and InterC terms, allowing joint optimization of phase predictions with temporal priors.
• Applies the regularization to frame-level probability maps, stabilizing predictions and suppressing false positives in ambiguous regions.
• Validates the method on THUMOS14 and ActivityNet1.3, showing consistent gains across multiple baselines and architectures.

Experimental results

Research questions

• RQ1 Can enforcing temporal smoothness within each action phase improve frame-level prediction consistency and localization accuracy in bottom-up TAL?
• RQ2 Does modeling the ordered relationship between starting, continuing, and ending phases reduce contradictory predictions and improve localization quality?
• RQ3 Can the proposed mutual regularization be effectively transferred to other TAL frameworks beyond the base model?
• RQ4 To what extent does the regularization improve performance under strict IoU thresholds, such as IoU=0.7?
• RQ5 How do the proposed consistency losses compare to alternative smoothing or regularization strategies in weakly-supervised or unsupervised settings?

Key findings

• The proposed method achieves a 6.8% absolute improvement in mAP at IoU=0.7 on the THUMOS14 dataset compared to the previous state-of-the-art.
• On ActivityNet1.3, the method improves mAP by 2.1% at IoU=0.3 and 1.8% at IoU=0.5 compared to the baseline.
• The IntraC and InterC regularizations significantly stabilize predictions, reducing false positives in starting and ending phase maps.
• Qualitative results show that the regularization suppresses spurious proposals, such as incorrect starting points between actions like "CleanAndJerk" and "Wakeboarding".
• The mutual regularization generalizes well: when applied to TSA-Net and PGCN, both models achieve improved performance, confirming the method's transferability.
• Oracle analysis reveals that proposal ranking is a major bottleneck, and the proposed method closes a significant portion of the gap by improving prediction consistency.