[Paper Review] Towards Automatic Learning of Procedures from Web Instructional Videos

Luowei Zhou, Chenliang Xu+1 more|arXiv (Cornell University)|Mar 28, 2017

Multimodal Machine Learning ApplicationsComputer Science222 citations

TL;DR

The paper defines procedure segmentation for unconstrained videos, introduces YouCook2 dataset, and presents ProcNets, a segment-level recurrent model that outperforms baselines in segmenting long instructional videos into category-independent procedure steps.

ABSTRACT

The potential for agents, whether embodied or software, to learn by observing other agents performing procedures involving objects and actions is rich. Current research on automatic procedure learning heavily relies on action labels or video subtitles, even during the evaluation phase, which makes them infeasible in real-world scenarios. This leads to our question: can the human-consensus structure of a procedure be learned from a large set of long, unconstrained videos (e.g., instructional videos from YouTube) with only visual evidence? To answer this question, we introduce the problem of procedure segmentation--to segment a video procedure into category-independent procedure segments. Given that no large-scale dataset is available for this problem, we collect a large-scale procedure segmentation dataset with procedure segments temporally localized and described; we use cooking videos and name the dataset YouCook2. We propose a segment-level recurrent network for generating procedure segments by modeling the dependencies across segments. The generated segments can be used as pre-processing for other tasks, such as dense video captioning and event parsing. We show in our experiments that the proposed model outperforms competitive baselines in procedure segmentation.

Motivation & Objective

• Motivate learning human-consensus procedure structure from long unconstrained instructional videos (e.g., YouTube).
• Define and tackle the procedure segmentation problem to partition videos into category-independent segments.
• Create a large-scale, richly annotated dataset to enable study of procedure segmentation (YouCook2).
• Develop an end-to-end model (ProcNets) that localizes segment proposals and learns segment-level temporal dependencies.
• Demonstrate that segment-level modeling improves over frame-level baselines and baselines without subtitles.

Proposed method

• Introduce context-aware frame encoding via ResNet features followed by a Bi-LSTM to produce context-aware frame representations.
• Propose a segment proposal module with K anchors (anchor-based proposals) to generate candidate procedure segments with start/end offsets, trained with binary classification and offset regression.
• Use a sequential prediction module (LSTM) that models segment-level dependencies to select and output the final sequence of procedure segments, incorporating Proposal Vector, Location Embedding, and Segment Content as inputs.
• Train with a combined loss L = L_cla + alpha_r L_reg + alpha_s L_seq, where L_cla is binary cross-entropy for procedureness, L_reg is smooth L1 for offsets, and L_seq is cross-entropy for sequential predictions.
• Infer with beam search to output a coherent sequence of procedure segments, without requiring a fixed number of segments.

Experimental results

Research questions

• RQ1 Can the human-consensus structure of a procedure be learned from long, unconstrained videos using only visual evidence?
• RQ2 Is a segment-level sequential model able to capture long-range dependencies among procedure steps better than frame-level approaches or non-sequential proposals?
• RQ3 Does a large-scale, richly annotated dataset enable robust learning and evaluation of category-independent procedure segmentation?
• RQ4 Can outputs of procedure segmentation improve downstream tasks like dense captioning or event parsing in instructional videos?

Key findings

• ProcNets significantly outperform competitive baselines on procedure segmentation in Jaccard and mIoU metrics (validation and test).
• ProcNets-LSTM achieves the highest scores: validation Jaccard 51.5, validation mIoU 37.5, test Jaccard 50.6, test mIoU 37.0.
• ProcNets-NMS improves over baselines that rely on non-maximum suppression alone, showing strong segment localization.
• Location Embedding is the most critical component for learning procedure structure, with notable drops when it is removed.
• The model adapts the number of segments per video and demonstrates qualitative understanding of procedural structure, including handling of non-annotated but semantically meaningful segments.
• YouCook2 dataset provides 2000 videos over 89 recipes with temporal procedure annotations and imperative sentences, enabling robust evaluation.