[Paper Review] Proposal-Based Multiple Instance Learning for Weakly-Supervised Temporal Action Localization

Huan Ren, Wenfei Yang+2 more|arXiv (Cornell University)|May 29, 2023

Human Pose and Action Recognition5 citations

TL;DR

This paper proposes a Proposal-based Multiple Instance Learning (P-MIL) framework for weakly-supervised temporal action localization that directly classifies candidate proposals in both training and inference, overcoming the inconsistency between segment-level training and proposal-level testing in existing S-MIL methods. The method introduces a surrounding contrastive feature extraction module, a proposal completeness evaluation module, and an instance-level rank consistency loss, achieving state-of-the-art performance with a 46.5% average mAP on THUMOS14 and ActivityNet.

ABSTRACT

Weakly-supervised temporal action localization aims to localize and recognize actions in untrimmed videos with only video-level category labels during training. Without instance-level annotations, most existing methods follow the Segment-based Multiple Instance Learning (S-MIL) framework, where the predictions of segments are supervised by the labels of videos. However, the objective for acquiring segment-level scores during training is not consistent with the target for acquiring proposal-level scores during testing, leading to suboptimal results. To deal with this problem, we propose a novel Proposal-based Multiple Instance Learning (P-MIL) framework that directly classifies the candidate proposals in both the training and testing stages, which includes three key designs: 1) a surrounding contrastive feature extraction module to suppress the discriminative short proposals by considering the surrounding contrastive information, 2) a proposal completeness evaluation module to inhibit the low-quality proposals with the guidance of the completeness pseudo labels, and 3) an instance-level rank consistency loss to achieve robust detection by leveraging the complementarity of RGB and FLOW modalities. Extensive experimental results on two challenging benchmarks including THUMOS14 and ActivityNet demonstrate the superior performance of our method.

Motivation & Objective

• To address the inconsistency between training (segment-level scoring) and testing (proposal-level scoring) objectives in existing S-MIL frameworks for weakly-supervised temporal action localization.
• To suppress overfitting to discriminative short proposals by leveraging surrounding contextual information during feature extraction.
• To improve detection robustness by refining low-quality proposals using completeness pseudo-labels and modality complementarity.
• To enhance relative scoring consistency among proposals of the same action instance through an instance-level rank consistency loss.
• To achieve unified training and testing objectives by directly classifying proposals throughout the pipeline.

Proposed method

• Proposes a two-stage training pipeline: first pre-train an S-MIL model to generate candidate proposals, then fine-tune a P-MIL model to directly classify these proposals.
• Introduces a Surrounding Contrastive Feature Extraction (SCFE) module that extends proposal boundaries and computes outer-inner contrastive features to suppress overconfident short proposals.
• Employs a Proposal Completeness Evaluation (PCE) module that uses high-confidence proposals as pseudo-instances to generate completeness pseudo-labels via IoU, inhibiting low-quality proposals.
• Designs an Instance-level Rank Consistency (IRC) loss that leverages RGB and FLOW modality complementarity to stabilize relative scoring within proposal clusters of the same action instance.
• Uses a unified loss function combining classification, completeness, and rank consistency objectives, with hyperparameters λcomp and λIRC for balancing.
• Incorporates additional background proposals during training to improve foreground-background separation, enhancing generalization.

Experimental results

Research questions

• RQ1 Can a P-MIL framework that directly classifies proposals unify training and testing objectives, leading to improved localization performance?
• RQ2 How effective is surrounding contrastive feature extraction in suppressing discriminative short proposals and improving feature representation?
• RQ3 To what extent does completeness pseudo-labeling enhance the quality of proposal scoring and detection accuracy?
• RQ4 Does enforcing instance-level rank consistency across proposals of the same action improve detection robustness and mAP?
• RQ5 How sensitive is the P-MIL model to hyperparameter settings, and can it maintain stable performance across different configurations?

Key findings

• The P-MIL framework achieves a 46.5% average mAP on the THUMOS14 and ActivityNet benchmarks, outperforming S-MIL baselines by 2.9% in mAP.
• The addition of background proposals during training improves mAP from 41.2% to 46.5%, demonstrating the benefit of enhanced foreground-background separation.
• The Surrounding Contrastive Feature Extraction (SCFE) module increases mAP by 5.5% compared to baseline feature extraction, validating its effectiveness in suppressing short, discriminative proposals.
• The Proposal Completeness Evaluation (PCE) module improves mAP by 0.7%, and the Instance-level Rank Consistency (IRC) loss by 0.8%, with combined gains of 1.3%.
• The model shows low sensitivity to hyperparameters λcomp and λirc, with mAP@0.5 fluctuations under 2% across different settings, indicating robustness.
• Fusing predictions from S-MIL and P-MIL models further boosts mAP to 47.0%, confirming complementary strengths between the two approaches.