[Paper Review] Multimodal Subtask Graph Generation from Instructional Videos

Yunseok Jang, Sungryull Sohn+4 more|arXiv (Cornell University)|Feb 17, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes MSG², a multimodal approach that generates subtask graphs from noisy instructional videos by integrating visual and textual signals to infer missing subtasks and model causal dependencies. It achieves state-of-the-art performance in next subtask prediction, outperforming video transformer models by 85% and 30% on ProceL and CrossTask datasets, respectively.

ABSTRACT

Real-world tasks consist of multiple inter-dependent subtasks (e.g., a dirty pan needs to be washed before it can be used for cooking). In this work, we aim to model the causal dependencies between such subtasks from instructional videos describing the task. This is a challenging problem since complete information about the world is often inaccessible from videos, which demands robust learning mechanisms to understand the causal structure of events. We present Multimodal Subtask Graph Generation (MSG2), an approach that constructs a Subtask Graph defining the dependency between a task's subtasks relevant to a task from noisy web videos. Graphs generated by our multimodal approach are closer to human-annotated graphs compared to prior approaches. MSG2 further performs the downstream task of next subtask prediction 85% and 30% more accurately than recent video transformer models in the ProceL and CrossTask datasets, respectively.

Motivation & Objective

• To model causal dependencies between subtasks in real-world tasks from multiple noisy instructional videos.
• To address data noise in web videos—such as missing, reordered, or partially annotated subtasks—through multimodal reasoning.
• To generate a more expressive subtask graph representation that captures complex dependencies (e.g., AND conditions) compared to prior methods.
• To improve downstream task performance, particularly next subtask prediction, using the inferred subtask graph structure.
• To develop a robust framework that generalizes across diverse video demonstrations and handles incomplete or inconsistent annotations.

Proposed method

• Uses multimodal subtask state prediction to infer the progress of each subtask using video frames and automatic speech recognition (ASR)-generated text.
• Applies a complexity-constrained inductive logic programming (ILP) method to infer subtask dependencies from incomplete and noisy subtask sequences.
• Constructs a subtask graph with expressive nodes (e.g., AND nodes) to model multiple preconditions and complex dependencies.
• Leverages multiple video instances of the same task to consolidate information and improve robustness to data noise.
• Integrates visual and linguistic signals to recover missing subtasks not present in annotations, such as omitted or unannotated steps.
• Employs a two-stage pipeline: first predicting subtask states per video, then generating a unified subtask graph from aggregated predictions.

Experimental results

Research questions

• RQ1 How can a model robustly infer causal subtask dependencies from multiple noisy, incomplete instructional videos?
• RQ2 To what extent can multimodal signals (visual and textual) improve subtask recovery and graph structure prediction in the presence of annotation errors?
• RQ3 Can a complexity-regularized ILP approach effectively generate accurate subtask graphs from noisy, incomplete subtask sequences?
• RQ4 How does the generated subtask graph improve downstream tasks like next subtask prediction compared to end-to-end video modeling?
• RQ5 What is the impact of using multiple video demonstrations on the robustness and accuracy of subtask graph generation?

Key findings

• MSG² generates subtask graphs that are significantly closer to human-annotated graphs than prior approaches, indicating improved structural accuracy.
• The method improves next subtask prediction accuracy by 85% over recent video transformer models on the ProceL dataset.
• On the CrossTask dataset, MSG² achieves a 30% improvement in next subtask prediction accuracy compared to state-of-the-art video transformers.
• The multimodal subtask state prediction module effectively recovers missing subtasks not present in annotations, enhancing graph completeness.
• The complexity-constrained ILP approach successfully handles noisy and incomplete data, producing more reliable and interpretable subtask graphs.
• The two-stage design—subtask state prediction followed by graph generation—enables better generalization and performance than end-to-end alternatives.