[Paper Review] A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos

Frank F. Xu, Lei Ji+5 more|arXiv (Cornell University)|May 2, 2020

Multimodal Machine Learning Applications55 references4 citations

TL;DR

This paper introduces a benchmark for structured procedural knowledge extraction from cooking videos, requiring models to generate verb-argument tuples from multimodal inputs (video and noisy transcripts). Despite using state-of-the-art models for action detection and semantic role labeling, performance remains low, highlighting the challenge of grounding actions in real-world procedures with coreference, ellipsis, and speech recognition errors.

ABSTRACT

Watching instructional videos are often used to learn about procedures. Video captioning is one way of automatically collecting such knowledge. However, it provides only an indirect, overall evaluation of multimodal models with no finer-grained quantitative measure of what they have learned. We propose instead, a benchmark of structured procedural knowledge extracted from cooking videos. This work is complementary to existing tasks, but requires models to produce interpretable structured knowledge in the form of verb-argument tuples. Our manually annotated open-vocabulary resource includes 356 instructional cooking videos and 15,523 video clip/sentence-level annotations. Our analysis shows that the proposed task is challenging and standard modeling approaches like unsupervised segmentation, semantic role labeling, and visual action detection perform poorly when forced to predict every action of a procedure in a structured form.

Motivation & Objective

• To address the lack of large-scale, open-vocabulary, structured procedural knowledge benchmarks for multimodal instructional videos.
• To create a dataset that enables fine-grained evaluation of models in extracting procedural steps as verb-argument tuples from video and transcript.
• To identify key challenges in procedural knowledge extraction, including verb ellipsis, speech recognition errors, and visual-linguistic coreference.
• To evaluate the limitations of current models in grounding actions across modalities, especially when transcripts are noisy or incomplete.

Proposed method

• The dataset comprises 356 cooking videos with 15,523 sentence-level annotations, manually labeled for procedural tuples (verb, arguments) grounded in video clips.
• A pipelined approach is used: first identify key video clips and sentences using multimodal segmentation, then extract tuples via semantic role labeling and visual action recognition.
• Semantic role labeling is applied to transcripts using a state-of-the-art model to extract verb-argument structures.
• Visual action detection models trained on kitchen videos are used to detect actions and objects, which are then aligned with textual predictions.
• Models are evaluated on their ability to predict full procedural tuples, including handling null arguments and coreference.
• The benchmark includes both supervised and unsupervised segmentation baselines to assess performance across different levels of supervision.

Experimental results

Research questions

• RQ1 How well can existing models extract structured procedural knowledge (verb-argument tuples) from multimodal cooking videos with noisy transcripts?
• RQ2 What are the main failure modes in procedural knowledge extraction, particularly regarding verb ellipsis and speech recognition errors?
• RQ3 To what extent do visual and linguistic signals need to be jointly modeled to accurately ground actions in video?
• RQ4 How do coreference and implicit arguments affect the performance of current models in procedural knowledge extraction?
• RQ5 Can multimodal models trained on weak supervision from instructional videos generalize to open-vocabulary procedural knowledge extraction?

Key findings

• Standard models for semantic role labeling and visual action detection perform poorly when forced to predict all procedural steps in structured form, indicating the task is highly challenging.
• Verb ellipsis is a major issue: models often misidentify the main verb (e.g., selecting 'give' instead of 'flip') due to lack of proper context-aware verb selection.
• Speech-to-text errors significantly degrade performance, with common errors like 'flour' vs. 'flower' or 'sriracha' vs. 'sarrah cha' leading to incorrect argument extraction.
• Coreference and implicit arguments remain unresolved in current baselines, with no explicit modeling of visual-linguistic co-reference.
• The proposed benchmark reveals that current models fail to generalize across diverse, real-world video transcripts with noise, highlighting the need for better cross-modal grounding.
• Even with state-of-the-art components, performance remains far from optimal, indicating that joint modeling of vision and language is essential for robust procedural knowledge extraction.