[Paper Review] Towards Train-Test Consistency for Semi-supervised Temporal Action Localization

Xudong Lin, Zheng Shou+1 more|arXiv (Cornell University)|Oct 24, 2019

Human Pose and Action Recognition64 references4 citations

TL;DR

This paper proposes TTC-Loc, a train-test consistent framework for semi-supervised temporal action localization that predicts adaptive thresholds at both training and inference time to eliminate the discrepancy between training (attention/MIL aggregation) and testing (fixed-threshold segmentation) strategies. By enabling explicit supervision from temporal boundary annotations during training, TTC-Loc achieves state-of-the-art performance, reaching 33.4% mAP (IoU=0.5) on THUMOS'14 with only one fully-annotated video per class.

ABSTRACT

Recently, Weakly-supervised Temporal Action Localization (WTAL) has been densely studied but there is still a large gap between weakly-supervised models and fully-supervised models. It is practical and intuitive to annotate temporal boundaries of a few examples and utilize them to help WTAL models better detect actions. However, the train-test discrepancy of action localization strategy prevents WTAL models from leveraging semi-supervision for further improvement. At training time, attention or multiple instance learning is used to aggregate predictions of each snippet for video-level classification; at test time, they first obtain action score sequences over time, then truncate segments of scores higher than a fixed threshold, and post-process action segments. The inconsistent strategy makes it hard to explicitly supervise the action localization model with temporal boundary annotations at training time. In this paper, we propose a Train-Test Consistent framework, TTC-Loc. In both training and testing time, our TTC-Loc localizes actions by comparing scores of action classes and predicted threshold, which enables it to be trained with semi-supervision. By fixing the train-test discrepancy, our TTC-Loc significantly outperforms the state-of-the-art performance on THUMOS'14, ActivityNet 1.2 and 1.3 when only video-level labels are provided for training. With full annotations of only one video per class and video-level labels for the other videos, our TTC-Loc further boosts the performance and achieves 33.4\% mAP (IoU threshold 0.5) on THUMOS's 14.

Motivation & Objective

• Address the large performance gap between weakly-supervised and fully-supervised temporal action localization by leveraging limited full annotations.
• Identify and resolve the train-test discrepancy in conventional WTAL methods, where different strategies are used for localization at train and test time.
• Enable explicit supervision of action localization models using temporal boundary annotations during training by aligning training and inference behavior.
• Propose a semi-supervised temporal action localization (STAL) setting that combines video-level labels and sparse full annotations for improved model generalization.
• Achieve state-of-the-art performance on benchmark datasets under both weakly-supervised and semi-supervised settings.

Proposed method

• Introduce a train-test consistent localization strategy that uses predicted thresholds for action segment detection at both training and inference time, replacing fixed manual thresholds.
• Design a differentiable gating mechanism using a sigmoid function to generate adaptive thresholds from video-level features, enabling end-to-end training.
• Formulate a regularization loss $L_{\text{reg}}$ that maximizes the margin between action scores and background scores via inner product, promoting robust action proposal generation.
• Integrate a localization loss $L_{\text{loc}}$ that directly supervises the model using temporal boundary annotations of a few videos, enabling semi-supervised learning.
• Use joint training on both video-level labels and partial full annotations, with a default strategy that outperforms pre-training and fine-tuning.
• Employ a unified loss function combining classification loss $L_{\text{clas}}$, regularization loss $L_{\text{reg}}$, and localization loss $L_{\text{loc}}$ for end-to-end optimization.

Experimental results

Research questions

• RQ1 Can eliminating the train-test discrepancy in action localization improve model performance under weakly-supervised settings?
• RQ2 How does predicting adaptive thresholds during training affect the model's ability to leverage temporal boundary annotations?
• RQ3 What is the optimal training strategy for combining video-level labels and sparse full annotations in semi-supervised temporal action localization?
• RQ4 Does the choice of gating function and regularization formulation significantly impact localization performance?
• RQ5 Can a consistent train-test strategy enable semi-supervised models to approach fully-supervised performance with minimal full annotations?

Key findings

• TTC-Loc achieves 32.0% mAP on THUMOS'14 under the semi-supervised setting (one fully-annotated video per class), significantly outperforming prior state-of-the-art methods.
• With only one fully-annotated video per class and video-level labels for the rest, TTC-Loc attains 33.4% mAP at IoU=0.5 on THUMOS'14, demonstrating strong generalization under minimal supervision.
• The joint training strategy with full and weak supervision outperforms pre-training and fine-tuning, achieving 32.0% mAP compared to 28.4% with the latter.
• The sigmoid gating function yields superior performance (28.2% mAP) compared to binarization (13.5%) and softsign (27.4%), indicating better optimization stability.
• The inner product formulation of $L_{\text{reg}}$ achieves the best performance (28.2% mAP), outperforming L1, L2, and cosine similarity due to stronger margin enforcement.
• Adding $L_{\text{reg}}$ and $L_{\text{loc}}$ significantly improves localization accuracy, with $L_{\text{reg}}$ alone boosting performance from 18.4% to 28.2% mAP.