[Paper Review] P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision

He Zhao, Isma Hadji+4 more|arXiv (Cornell University)|May 4, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes P3IV, a weakly supervised, probabilistic procedure planning framework that leverages pre-trained text-video embeddings for language supervision instead of costly temporal video annotations. It uses a single-branch transformer with a memory module and a probabilistic generative module to generate diverse, high-quality action sequences in parallel, outperforming fully supervised state-of-the-art models on multiple metrics by effectively modeling uncertainty in planning.

ABSTRACT

In this paper, we study the problem of procedure planning in instructional videos. Here, an agent must produce a plausible sequence of actions that can transform the environment from a given start to a desired goal state. When learning procedure planning from instructional videos, most recent work leverages intermediate visual observations as supervision, which requires expensive annotation efforts to localize precisely all the instructional steps in training videos. In contrast, we remove the need for expensive temporal video annotations and propose a weakly supervised approach by learning from natural language instructions. Our model is based on a transformer equipped with a memory module, which maps the start and goal observations to a sequence of plausible actions. Furthermore, we augment our model with a probabilistic generative module to capture the uncertainty inherent to procedure planning, an aspect largely overlooked by previous work. We evaluate our model on three datasets and show our weaklysupervised approach outperforms previous fully supervised state-of-the-art models on multiple metrics.

Motivation & Objective

• To address the challenge of high annotation cost in supervised procedure planning from instructional videos.
• To reduce reliance on expensive temporal video annotations by leveraging natural language instructions as weak supervision.
• To model uncertainty in procedure planning through a probabilistic generative module, enabling multiple feasible plans.
• To improve planning performance by using a non-autoregressive, single-branch transformer architecture that generates all steps in parallel.
• To evaluate the quality and diversity of predicted plans using metrics like KL divergence, NLL, ModeRec, and ModePrec.

Proposed method

• The model uses a transformer-based decoder with a memory module to map start and goal visual observations to a sequence of actions.
• It replaces expensive intermediate visual annotations with language supervision via pre-trained text-video embeddings of instructional steps.
• A probabilistic generative module samples K=1500 action sequences to model uncertainty and capture multiple plausible plans.
• The model is trained using contrastive learning on paired visual and linguistic representations, with plan quality assessed via KL divergence and NLL.
• Plan diversity and coverage are evaluated using Mode Recall (ModeRec) and Mode Precision (ModePrec), with cosine distance used to measure sample diversity.
• The approach uses a one-shot inference mechanism, generating all intermediate actions simultaneously rather than autoregressively.

Experimental results

Research questions

• RQ1 Can we achieve state-of-the-art procedure planning performance without requiring expensive temporal annotations of intermediate steps in instructional videos?
• RQ2 Does leveraging pre-trained text-video embeddings as weak supervision lead to better generalization and planning quality than strong visual supervision?
• RQ3 Can a non-autoregressive, single-branch transformer model effectively generate diverse and accurate action sequences in parallel?
• RQ4 How does modeling uncertainty through probabilistic sampling improve plan diversity and coverage compared to deterministic planning?
• RQ5 Can metrics like KL divergence, NLL, ModeRec, and ModePrec effectively evaluate the quality and distributional alignment of predicted plans?

Key findings

• Our probabilistic approach achieves lower KL divergence (2.11 vs. 2.31 at T=3) and NLL (4.89 vs. 5.13 at T=3) than the deterministic variant, indicating better alignment with ground-truth plan distributions.
• The probabilistic model improves Mode Recall by 9.89% at T=3 and Mode Precision by 9.00% at T=3, showing better coverage and accuracy of ground-truth modes.
• Cosine distance among samples is 0.384 at T=3, indicating high diversity in the generated plans, which is not achievable with the deterministic model.
• The model outperforms fully supervised SOTA methods on multiple metrics despite using only weak language supervision, demonstrating the effectiveness of the proposed approach.
• The results confirm that modeling uncertainty via probabilistic sampling leads to more robust and realistic planning, especially in scenarios with multiple valid action sequences.
• The use of pre-trained text-video embeddings as supervision significantly reduces annotation cost while maintaining or improving performance.