[Paper Review] Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos

Muheng Li, Lei Chen+5 more|arXiv (Cornell University)|Mar 26, 2022

Human Pose and Action Recognition4 citations

TL;DR

Bridge-Prompt proposes a prompt-based framework that models semantic relationships across ordinal actions in instructional videos by integrating textual prompts that encode both individual action semantics and their sequential order. By jointly training vision and text encoders via contrastive learning on these structured prompts, the method achieves state-of-the-art performance on action segmentation and recognition benchmarks such as GTEA, 50Salads, and Breakfast, demonstrating improved contextual understanding and zero-shot transferability.

ABSTRACT

Action recognition models have shown a promising capability to classify human actions in short video clips. In a real scenario, multiple correlated human actions commonly occur in particular orders, forming semantically meaningful human activities. Conventional action recognition approaches focus on analyzing single actions. However, they fail to fully reason about the contextual relations between adjacent actions, which provide potential temporal logic for understanding long videos. In this paper, we propose a prompt-based framework, Bridge-Prompt (Br-Prompt), to model the semantics across adjacent actions, so that it simultaneously exploits both out-of-context and contextual information from a series of ordinal actions in instructional videos. More specifically, we reformulate the individual action labels as integrated text prompts for supervision, which bridge the gap between individual action semantics. The generated text prompts are paired with corresponding video clips, and together co-train the text encoder and the video encoder via a contrastive approach. The learned vision encoder has a stronger capability for ordinal-action-related downstream tasks, e.g. action segmentation and human activity recognition. We evaluate the performances of our approach on several video datasets: Georgia Tech Egocentric Activities (GTEA), 50Salads, and the Breakfast dataset. Br-Prompt achieves state-of-the-art on multiple benchmarks. Code is available at https://github.com/ttlmh/Bridge-Prompt

Motivation & Objective

• To address the limitation of conventional action recognition models that fail to capture contextual relationships between adjacent actions in long instructional videos.
• To model the semantics of ordinal action sequences by leveraging natural language's ability to express temporal order and relational semantics.
• To develop a prompt-based learning framework that jointly trains vision and text encoders to exploit both out-of-context and contextual action information.
• To improve performance on downstream tasks such as action segmentation and long-term human activity recognition in instructional video analysis.
• To enable zero-shot transferability between similar activities by leveraging the generalization capacity of pre-trained language models.

Proposed method

• Design a three-plus-one-level text prompt structure that integrates statistical, ordinal, and semantic information into unified textual supervision for sequences of actions.
• Construct a video-text fusion module that cross-attentionally aligns video features with the corresponding text prompts during joint training.
• Train the vision and text encoders end-to-end using a contrastive learning objective that aligns video and prompt representations based on their semantic and sequential consistency.
• Incorporate three loss components: semantic loss ($\mathcal{L}_{\text{sem}}$), integrated semantics loss ($\mathcal{L}_{\text{integ}}$), and statistical loss ($\mathcal{L}_{\text{stat}}$) to jointly optimize for accuracy and robustness.
• Utilize pre-trained vision and language models (e.g., ViT and BERT) as encoders to benefit from large-scale pre-training and enable effective few- and zero-shot generalization.
• Apply prompt engineering to enable zero-shot inference on unseen action types by modifying only the textual prompt, without retraining.

Experimental results

Research questions

• RQ1 Can structured text prompts effectively model the semantic and sequential relationships between adjacent actions in instructional videos?
• RQ2 How does the integration of ordinal and contextual semantics in prompts improve action recognition and segmentation beyond single-action classification?
• RQ3 To what extent can the Bridge-Prompt framework generalize to unseen action types or similar activities through prompt-based zero-shot transfer?
• RQ4 What is the contribution of different loss components (semantic, integrated, statistical) to the overall performance of the framework?
• RQ5 How does Bridge-Prompt compare to graph-based and class-ID-based methods in terms of scalability, generalization, and performance on benchmark datasets?

Key findings

• Bridge-Prompt achieves state-of-the-art performance on the GTEA dataset, with an F1@50 score of 91.0 and edit score of 89.6, outperforming prior methods.
• The ablation study confirms that all three loss components—semantic, integrated semantics, and statistical—contribute positively to performance, with the full loss combination yielding the best results.
• The framework demonstrates strong zero-shot transferability: for example, transferring knowledge from coffee-making to tea-making yields a top-1 accuracy of 41.7% and top-5 accuracy of 81.3% on GTEA.
• The method significantly outperforms conventional class-ID-based approaches in modeling contextual action relations, as illustrated in Figure 1, where Bridge-Prompt captures inter-action semantics that individual labels miss.
• The use of natural language prompts enables effective generalization to unseen action types, unlike rigid class-ID systems that require manual re-labeling for new actions.
• The contrastive learning strategy with carefully designed prompts enables the vision encoder to learn richer, more contextual representations, enhancing downstream task performance across multiple benchmarks.