[Paper Review] Weakly Supervised Temporal Adjacent Network for Language Grounding

Yuechen Wang, Jiajun Deng+2 more|arXiv (Cornell University)|Jun 30, 2021

Multimodal Machine Learning Applications62 references11 citations

TL;DR

This paper proposes Weakly Supervised Temporal Adjacent Network (WSTAN), a novel framework for temporal language grounding that formulates the task as a multiple instance learning (MIL) problem using a full paragraph of descriptions instead of individual queries. By leveraging a temporal adjacent network, a complementary branch with pseudo supervision, and a self-discriminating loss, WSTAN improves cross-modal alignment and temporal precision, achieving state-of-the-art performance on three benchmark datasets without requiring temporal boundary annotations.

ABSTRACT

Temporal language grounding (TLG) is a fundamental and challenging problem for vision and language understanding. Existing methods mainly focus on fully supervised setting with temporal boundary labels for training, which, however, suffers expensive cost of annotation. In this work, we are dedicated to weakly supervised TLG, where multiple description sentences are given to an untrimmed video without temporal boundary labels. In this task, it is critical to learn a strong cross-modal semantic alignment between sentence semantics and visual content. To this end, we introduce a novel weakly supervised temporal adjacent network (WSTAN) for temporal language grounding. Specifically, WSTAN learns cross-modal semantic alignment by exploiting temporal adjacent network in a multiple instance learning (MIL) paradigm, with a whole description paragraph as input. Moreover, we integrate a complementary branch into the framework, which explicitly refines the predictions with pseudo supervision from the MIL stage. An additional self-discriminating loss is devised on both the MIL branch and the complementary branch, aiming to enhance semantic discrimination by self-supervising. Extensive experiments are conducted on three widely used benchmark datasets, \emph{i.e.}, ActivityNet-Captions, Charades-STA, and DiDeMo, and the results demonstrate the effectiveness of our approach.

Motivation & Objective

• To address the high annotation cost of fully supervised temporal language grounding by enabling training with only video-level descriptions and no temporal boundary labels.
• To improve cross-modal semantic alignment between text and video content in weakly supervised settings, where traditional methods lack temporal discriminability.
• To enhance localization precision by modeling temporal context and refining predictions through a complementary branch with pseudo supervision.
• To explore the feasibility of using coarse video scripts as weak supervision, extending applicability to real-world data with minimal annotations.

Proposed method

• Formulates weakly supervised temporal language grounding as a multiple instance learning (MIL) problem, treating candidate video segments as bags and text paragraphs as instances.
• Constructs a 2D score map from evenly divided video clips, where each element represents a candidate temporal segment from clip i to j, enabling modeling of temporal relationships.
• Employs a temporal adjacent network within the MIL framework to exploit sequential dependencies among video segments and improve alignment with paragraph-level text input.
• Introduces a complementary branch that refines predictions from the MIL branch using pseudo labels generated from intermediate results, reducing semantic sparsity.
• Applies a self-discriminating loss on both the MIL and complementary branches to enhance temporal discriminability by emphasizing semantically rich video segments.
• Uses thresholded pseudo labels (o_max=1.0, o_min=0.9) to reduce uncertainty in supervision signals during training.

Experimental results

Research questions

• RQ1 Can paragraph-level text input improve cross-modal alignment and localization performance in weakly supervised temporal language grounding compared to single-query inputs?
• RQ2 How effective is the complementary branch in refining MIL predictions and uncovering semantically meaningful clips that are not distinct in initial matching?
• RQ3 To what extent does the self-discriminating loss enhance temporal precision by emphasizing key semantic content such as small objects and postures?
• RQ4 Can WSTAN generalize to even weaker supervision, such as video-level scripts with complex syntax, without significant performance degradation?

Key findings

• WSTAN achieves state-of-the-art performance on three benchmark datasets—Charades-STA, ActivityNet-Captions, and DiDeMo—without requiring temporal boundary annotations.
• The paragraph input design improves performance on most metrics, demonstrating that aggregated text input reduces noise from semantically partial similarities across videos.
• The complementary branch significantly improves localization precision, as shown in qualitative results where the full WSTAN model detects subtle semantic concepts like 'off' that the base model misses.
• The self-discriminating loss enhances model discrimination, particularly improving performance on R@1, while the pseudo label threshold for the complementary branch mainly affects R@5.
• WSTAN generalizes well to video scripts as weak supervision, achieving comparable performance to standard description inputs, validating its robustness in practical, low-resource settings.