[Paper Review] TutorialVQA: Question Answering Dataset for Tutorial Videos

Anthony Colas, Seokhwan Kim+4 more|arXiv (Cornell University)|Dec 2, 2019

Multimodal Machine Learning Applications17 references4 citations

TL;DR

This paper introduces TutorialVQA, a new video question answering dataset comprising ~6,000 manually curated (video, question, answer span) triples from screencast tutorial videos for an image editing tool. The task focuses on localizing multi-step, non-factoid answers as video spans rather than short text answers, and baseline results show low performance, indicating the task is challenging and necessitates new models for accurate span localization in instructional videos.

ABSTRACT

Despite the number of currently available datasets on video question answering, there still remains a need for a dataset involving multi-step and non-factoid answers. Moreover, relying on video transcripts remains an under-explored topic. To adequately address this, We propose a new question answering task on instructional videos, because of their verbose and narrative nature. While previous studies on video question answering have focused on generating a short text as an answer, given a question and video clip, our task aims to identify a span of a video segment as an answer which contains instructional details with various granularities. This work focuses on screencast tutorial videos pertaining to an image editing program. We introduce a dataset, TutorialVQA, consisting of about 6,000manually collected triples of (video, question, answer span). We also provide experimental results with several baselines algorithms using the video transcripts. The results indicate that the task is challenging and call for the investigation of new algorithms.

Motivation & Objective

• To address the gap in video question answering datasets that support non-factoid, multi-step questions requiring span-based answers rather than short text responses.
• To create a dataset focused on instructional screencast videos where answers are distributed across multiple segments and require granular localization.
• To explore the feasibility of using video transcripts for answer span localization in long-form tutorial videos, which are underutilized in existing VQA benchmarks.
• To evaluate baseline models on both sentence-level prediction and video segment retrieval tasks, revealing significant challenges in span localization.
• To motivate future research by demonstrating that current models struggle with precise span localization, even when the correct video is identified.

Proposed method

• The dataset, TutorialVQA, is constructed from 6,000 manually collected triples of video clips, natural language questions, and corresponding answer spans within the video.
• Answer spans are defined as contiguous segments of the video that contain the complete answer to a non-factoid 'how-to' question, with variable granularity.
• Baseline models use TF-IDF vectorization to encode video transcripts, questions, and candidate segments, followed by cosine similarity to identify the most relevant segments.
• The first baseline performs sentence-level prediction with a tolerance window to account for minor boundary errors in span localization.
• The second baseline performs video segment retrieval by filtering top-10 most similar videos based on TF-IDF cosine similarity, then selecting the best segment within each.
• The third baseline combines video filtering and segment selection, evaluating performance both overall and conditional on the correct video being in the top 10.

Experimental results

Research questions

• RQ1 Can existing video question answering models effectively localize multi-step, non-factoid answers as spans in instructional tutorial videos?
• RQ2 How effective is using only video transcripts for span-based answer localization in long-form tutorial videos?
• RQ3 Does reducing the search space by first filtering relevant videos improve the accuracy of segment retrieval?
• RQ4 Why do current models fail to precisely locate answer boundaries despite identifying the correct region?
• RQ5 What architectural improvements, such as pointer networks, could enhance span boundary prediction accuracy?

Key findings

• The first baseline, using a tolerance window for span prediction, achieves only 14.43% accuracy with a window size of 6, indicating poor boundary localization.
• The second baseline, which performs video segment retrieval, achieves 23.41% accuracy on the sentence-level prediction task, showing limited effectiveness.
• When the correct video is within the top 10 filtered videos, the segment retrieval accuracy increases to 63.85%, suggesting that video filtering significantly improves performance.
• Error analysis reveals that 92% of incorrect predictions overlap with the ground-truth span, but are imprecise in boundary selection, indicating a need for better boundary modeling.
• The search space for span prediction is reduced from n² to 2n using a pointer network, which may improve accuracy by focusing on boundary sentences rather than all possible spans.
• The results demonstrate that current models are insufficient for this task, highlighting the need for new architectures and multimodal approaches that integrate visual and textual signals.