Skip to main content
QUICK REVIEW

[论文解读] EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding

Karttikeya Mangalam, Raiymbek Akshulakov|arXiv (Cornell University)|Aug 17, 2023
Multimodal Machine Learning ApplicationsComputer Science被引用 19
一句话总结

EgoSchema 提供一个来自 Ego4D 的非常长篇的视频问答基准,包含 5000+ 选择题,覆盖 250 小时的第一人称视频,并且分析使用时序证书集的内在时序难度。

ABSTRACT

We introduce EgoSchema, a very long-form video question-answering dataset, and benchmark to evaluate long video understanding capabilities of modern vision and language systems. Derived from Ego4D, EgoSchema consists of over 5000 human curated multiple choice question answer pairs, spanning over 250 hours of real video data, covering a very broad range of natural human activity and behavior. For each question, EgoSchema requires the correct answer to be selected between five given options based on a three-minute-long video clip. While some prior works have proposed video datasets with long clip lengths, we posit that merely the length of the video clip does not truly capture the temporal difficulty of the video task that is being considered. To remedy this, we introduce temporal certificate sets, a general notion for capturing the intrinsic temporal understanding length associated with a broad range of video understanding tasks & datasets. Based on this metric, we find EgoSchema to have intrinsic temporal lengths over 5.7x longer than the second closest dataset and 10x to 100x longer than any other video understanding dataset. Further, our evaluation of several current state-of-the-art video and language models shows them to be severely lacking in long-term video understanding capabilities. Even models with several billions of parameters achieve QA accuracy less than 33% (random is 20%) on the EgoSchema multi-choice question answering task, while humans achieve about 76% accuracy. We posit that ame{}{}, with its long intrinsic temporal structures and diverse complexity, would serve as a valuable evaluation probe for developing effective long-term video understanding systems in the future. Data and Zero-shot model evaluation code are open-sourced for both public and commercial use under the Ego4D license at http://egoschema.github.io

研究动机与目标

  • 介绍 EgoSchema,一个用于非常长篇视频-语言理解的诊断基准。
  • 定义时序证书长度,以捕捉超越片段长度的内在时序难度。
  • 证明当前最先进模型在非常长篇的问答任务上不及人类。

提出的方法

  • 通过筛选含有丰富叙述的 3 分钟 Ego4D 片段,使用 LLM 提示生成 QA 三元组(QAW),从而构建 EgoSchema。
  • 使用多种提示策略(QAW-shot,Q(AW)-shot)来创建问题和干扰项,聚焦于长期推理。
  • 应用基于规则和基于LLM的筛选,去除低质量或非证实的问题,并进行两轮人工策划,确保最小证书长度为 30 秒。
  • 将时序证书定义为验证正确答案所需的最小子片段,并计算每个片段的证书长度。
  • 基准测试多个视频-语言模型和人类在 EgoSchema 上的零-shot 问答表现。
Figure 1: The EgoSchema dataset contains over 5000 very long-form video language understanding questions spanning over 250 hours of real, diverse, and high-quality egocentric video data. Each question requires choosing the correct answer out of five choices based on a three minute long video clip. T
Figure 1: The EgoSchema dataset contains over 5000 very long-form video language understanding questions spanning over 250 hours of real, diverse, and high-quality egocentric video data. Each question requires choosing the correct answer out of five choices based on a three minute long video clip. T

实验结果

研究问题

  • RQ1通过时序证书长度衡量的 EgoSchema 的内在时序难度是多少?
  • RQ2现有前沿视频-语言模型在 EgoSchema 的零-shot、长时问答任务中的表现如何?
  • RQ3在不同时间/帧约束下,人与模型在极长篇视频问答中的表现有何差异?
  • RQ4相较于人类,当前模型在长期视频理解方面是否还有提升空间?

主要发现

  • EgoSchema 的中位时序证书长度约为 100 秒,大约是第二长数据集的 5 倍,且比其他视频理解数据集长 10 倍到 100 倍。
  • 亿级参数模型在 EgoSchema 的零-shot 问答准确率低于 33%,而在人类在无限制设置下约为 76%。
  • 一些被评估的模型表现出随着帧数增加而提升的准确率,但在约 30 帧时达到饱和,且仍远低于人类表现。
  • 在严格时间限制下,即使只有 1 fps 的视频,人类的准确率也约为 67%,在视频到文本设置下约为 76%,凸显与模型之间的差距。
  • EgoSchema 将在 Ego4D 许可下公开发布,以便进行研究和商业用途。
Figure 2: We introduce the notion of a temporal certificate set (top, § 3.2 ), a tool to measure the intrinsic temporal length of a benchmark and show the EgoSchema certificate length distribution (bottom, § 4.1 ) for randomly chosen $100$ clips.
Figure 2: We introduce the notion of a temporal certificate set (top, § 3.2 ), a tool to measure the intrinsic temporal length of a benchmark and show the EgoSchema certificate length distribution (bottom, § 4.1 ) for randomly chosen $100$ clips.

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。