[论文解读] FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation
FETV 引入一种细粒度、多方面、具有时间感知的开放域文本到视频(Text-to-Video, T2V)基准,并分析自动评估指标对人类判断的可靠性,提出基于 UMT 的改进指标。
Recently, open-domain text-to-video (T2V) generation models have made remarkable progress. However, the promising results are mainly shown by the qualitative cases of generated videos, while the quantitative evaluation of T2V models still faces two critical problems. Firstly, existing studies lack fine-grained evaluation of T2V models on different categories of text prompts. Although some benchmarks have categorized the prompts, their categorization either only focuses on a single aspect or fails to consider the temporal information in video generation. Secondly, it is unclear whether the automatic evaluation metrics are consistent with human standards. To address these problems, we propose FETV, a benchmark for Fine-grained Evaluation of Text-to-Video generation. FETV is multi-aspect, categorizing the prompts based on three orthogonal aspects: the major content, the attributes to control and the prompt complexity. FETV is also temporal-aware, which introduces several temporal categories tailored for video generation. Based on FETV, we conduct comprehensive manual evaluations of four representative T2V models, revealing their pros and cons on different categories of prompts from different aspects. We also extend FETV as a testbed to evaluate the reliability of automatic T2V metrics. The multi-aspect categorization of FETV enables fine-grained analysis of the metrics' reliability in different scenarios. We find that existing automatic metrics (e.g., CLIPScore and FVD) correlate poorly with human evaluation. To address this problem, we explore several solutions to improve CLIPScore and FVD, and develop two automatic metrics that exhibit significant higher correlation with humans than existing metrics. Benchmark page: https://github.com/llyx97/FETV.
研究动机与目标
- 为跨多种提示类别的开放域 T2V 模型提供细粒度评估框架。
- 引入时间感知以捕捉 T2V 生成中的视频特定挑战。
- 评估现有自动评估指标相对于人类判断的可靠性。
- 提供与人类评估对齐的改进自动评估指标,使用先进的视觉-语言模型。
提出的方法
- 提出 FETV,这是一个按主要内容、属性控制和提示复杂度对提示进行分类的基准,以及用于视频生成的时间维度类别。
- 从 MSR-VTT、WebVid 收集 619 条按类别的提示,并由人工撰写的异常提示;先进行自动标签,然后再进行人工标注以确定类别。
- 对四个开放 T2V 模型在四个维度进行人工评估(静态质量、时间质量、总体对齐、细粒度对齐),每个视频/模型由 3 名人工评审打分。

实验结果
研究问题
- RQ1开放域 T2V 模型在内容、属性和复杂度的细粒度类别上表现如何?
- RQ2时间内容如何影响视频质量及其与提示的对齐?
- RQ3自动 T2V 评估指标与人类判断之间的相关性如何,能否改进?
- RQ4基于先进视觉-语言模型的新评估指标是否能在各类别中更好地与人类评估保持一致?
- RQ5在多样化提示下,具有代表性的开放 T2V 模型的优缺点是什么?
主要发现
- 动作/运动的动作性以及在空间内容中的人/动物的视频质量更差。
- 模型在控制数量、运动方向和事件顺序等方面能力存在差异;当前模型在这些属性上存在困难。
- 自动评估指标 CLIPScore 和 FVD 与人类评估相关性较差;UMTScore 和 FVD-UMT 与人类相关性较高。
- 对齐更好于人类排名的微调或基于先进 VLM 的指标(UMTScore、FVD-UMT)比传统指标表现更好。
- 人工评估揭示 CogVideo、Text2Video-zero、ModelScopeT2V、ZeroScope 在各类别下的实际优缺点。

阅读本论文
在 Nubint 阅读器中打开原文,边读边就任意段落提问。
在 Nubint 中阅读本论文本解读由 AI 根据论文原文生成,引用前请核对原文。