[Paper Review] Learning and Verification of Task Structure in Instructional Videos

Medhini Narasimhan, Licheng Yu+3 more|arXiv (Cornell University)|Mar 23, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces VideoTaskformer, a video model that learns contextualized step representations in instructional videos by masking and predicting steps using global task context. It achieves state-of-the-art performance on downstream tasks, especially in mistake detection and long-term forecasting, and introduces three new benchmarks for evaluating task structure understanding in videos.

ABSTRACT

Given the enormous number of instructional videos available online, learning a diverse array of multi-step task models from videos is an appealing goal. We introduce a new pre-trained video model, VideoTaskformer, focused on representing the semantics and structure of instructional videos. We pre-train VideoTaskformer using a simple and effective objective: predicting weakly supervised textual labels for steps that are randomly masked out from an instructional video (masked step modeling). Compared to prior work which learns step representations locally, our approach involves learning them globally, leveraging video of the entire surrounding task as context. From these learned representations, we can verify if an unseen video correctly executes a given task, as well as forecast which steps are likely to be taken after a given step. We introduce two new benchmarks for detecting mistakes in instructional videos, to verify if there is an anomalous step and if steps are executed in the right order. We also introduce a long-term forecasting benchmark, where the goal is to predict long-range future steps from a given step. Our method outperforms previous baselines on these tasks, and we believe the tasks will be a valuable way for the community to measure the quality of step representations. Additionally, we evaluate VideoTaskformer on 3 existing benchmarks -- procedural activity recognition, step classification, and step forecasting -- and demonstrate on each that our method outperforms existing baselines and achieves new state-of-the-art performance.

Motivation & Objective

• To learn step representations in instructional videos that capture global task structure rather than local clip-level semantics.
• To enable verification of whether a video correctly performs a task, including detecting incorrect steps and wrong step ordering.
• To forecast long-range future steps from a given point in a video, improving on prior short-term forecasting methods.
• To introduce new benchmarks for evaluating the quality of step representations in instructional videos.
• To demonstrate that global context-aware representations outperform local, clip-independent methods on multiple downstream tasks.

Proposed method

• Pre-trains VideoTaskformer using a masked step modeling objective, where random steps in a video are masked and the model predicts their labels from visual features of the remaining steps.
• Uses a transformer-based architecture to jointly encode all steps in a video, enabling contextual understanding across the entire task.
• Leverages video features from pre-trained models (e.g., TimeSformer, S3D) and fine-tunes the full network end-to-end on downstream tasks.
• Incorporates task labels (e.g., 'Install a Ceiling Fan') as input embeddings to improve performance on challenging tasks like mistake detection and long-term forecasting.
• Evaluates both linear-probe and full fine-tuning settings to assess transferability of learned representations.
• Synthesizes a mistake detection dataset by replacing steps with similar ones or reordering steps from the COIN dataset.

Figure 1: Prior work [ 13 , 12 ] learns step representations from single short video clips, independent of the task, thus lacking knowledge of task structure. Our model, VideoTaskformer, learns step representations for masked video steps through the global context of all surrounding steps in the vid

Experimental results

Research questions

• RQ1 Can masked modeling of steps using global video context improve step representation learning in instructional videos compared to local clip-level modeling?
• RQ2 Can the learned representations detect both incorrect steps and incorrect step ordering in instructional videos?
• RQ3 How well can the model forecast long-range future steps from a given step in a task?
• RQ4 Does incorporating task-level labels improve performance on downstream tasks like mistake detection and forecasting?
• RQ5 To what extent do the representations transfer to existing benchmarks such as procedural activity recognition, step classification, and step forecasting?

Key findings

• VideoTaskformer achieves a 5% improvement over the best baseline on the long-term forecasting task, demonstrating strong generalization to distant future steps.
• It improves by 4% on mistake step detection and 4% on mistake ordering detection, outperforming prior methods on these novel benchmarks.
• On the procedural activity recognition task, VideoTaskformer achieves a 1% improvement over the best existing baseline, with distribution matching loss slightly outperforming step classification loss.
• Linear-probe evaluation shows competitive performance, indicating that the pre-trained representations are highly transferable to downstream tasks.
• Incorporating task labels boosts performance across all three new benchmarks, highlighting the value of high-level task context.
• Qualitative results confirm that VideoTaskformer correctly identifies incorrect steps and predicts future steps that match ground truth, while the baseline LwDS fails on complex structural reasoning.

Figure 2: VideoTaskformer Pre-training (Left). VideoTaskformer $f_{\text{VT}}$ learns step representations for the masked out video clip $v_{i}$ , while attending to the other clips in the video. It consists of a video encoder $f_{\text{vid}}$ , a step transformer $f_{\text{trans}}$ , and a linear l