[论文解读] Learning Situation Hyper-Graphs for Video Question Answering

Aisha Urooj Khan, Hilde Kuehne+6 more|arXiv (Cornell University)|Apr 18, 2023

Multimodal Machine Learning Applications被引用 4

一句话总结

本文提出 SHG-VQA，一种新颖的端到端视频问答框架，通过从视频片段中隐式学习情境超图（表示参与者、物体、关系与动作）来实现，无需显式的目标检测。通过使用情境超图解码器预测图标记，并利用问题嵌入进行交叉注意力，该模型在 AGQA 和 STAR 基准测试中达到最先进性能，表明隐式超图学习能显著提升对复杂视频内容的推理能力。

ABSTRACT

Answering questions about complex situations in videos requires not only capturing the presence of actors, objects, and their relations but also the evolution of these relationships over time. A situation hyper-graph is a representation that describes situations as scene sub-graphs for video frames and hyper-edges for connected sub-graphs and has been proposed to capture all such information in a compact structured form. In this work, we propose an architecture for Video Question Answering (VQA) that enables answering questions related to video content by predicting situation hyper-graphs, coined Situation Hyper-Graph based Video Question Answering (SHG-VQA). To this end, we train a situation hyper-graph decoder to implicitly identify graph representations with actions and object/human-object relationships from the input video clip. and to use cross-attention between the predicted situation hyper-graphs and the question embedding to predict the correct answer. The proposed method is trained in an end-to-end manner and optimized by a VQA loss with the cross-entropy function and a Hungarian matching loss for the situation graph prediction. The effectiveness of the proposed architecture is extensively evaluated on two challenging benchmarks: AGQA and STAR. Our results show that learning the underlying situation hyper-graphs helps the system to significantly improve its performance for novel challenges of video question-answering tasks.

研究动机与目标

• 解决在复杂、动态场景中进行视频问答所面临的挑战，尤其需要时空推理能力。
• 通过将情境建模为编码实体、关系及随时间演化的动作的超图，提升 VQA 中的推理能力。
• 从视频输入中隐式学习结构化的场景表征，无需依赖预训练目标检测器或显式图计算。
• 通过多任务损失联合优化模型，以实现准确的情境超图预测与正确答案生成。
• 在具有密集真实超图标注的挑战性基准上，评估超图学习的有效性。

提出的方法

• 引入情境超图解码器，从视频片段特征中以标记集合的形式预测原子动作及物体/参与者-物体间关系。
• 模型通过预测的超图标记与问题嵌入之间的交叉注意力生成答案。
• 在每一帧上应用二分图匹配损失（匈牙利匹配），以对齐预测与真实超图标记，提升集合预测的准确性。
• 整体训练目标结合了 VQA 损失（交叉熵）与匈牙利匹配损失，用于动作与关系预测。
• 该架构为端到端可训练，推理过程中无需显式图神经网络层。
• 该方法将时空图学习视为集合预测任务，实现对关系与动作的直接预测，无需中间图构建过程。

Figure 1 : The situation hyper-graph for a video is composed of situations with entities and their relationships (shown as subgraphs in the pink box). These situations may evolve over time. Temporal actions act as hyper-edges connecting these situations into one situation hyper-graph. Learning situa

实验结果

研究问题

• RQ1 从原始视频中隐式学习情境超图是否能提升视频问答性能？
• RQ2 将动作建模为连接子图的超边，是否能增强对动态视频内容的推理能力？
• RQ3 与全视频匹配相比，帧级匈牙利匹配在超图预测中的准确性和时间一致性表现如何？
• RQ4 学习结构化的超图表征在分布外推理任务中的泛化能力提升程度如何？
• RQ5 基于 Transformer 的架构是否能在无需显式图计算或预定义场景图头的情况下实现 VQA 的竞争力表现？

主要发现

• 所提出的 SHG-VQA 模型在 AGQA 基准上达到最先进性能，使用 4 个动作和 12 个关系时准确率达到 39.90%。
• 在 STAR 基准上，模型在使用 3 个动作和 8 个关系时达到 39.20% 的准确率，展现出在复杂推理类型上的强大性能。
• 消融实验表明，高质量的超图预测对 VQA 性能至关重要，图生成质量差的模型无法回答复杂问题。
• 与全视频匹配相比，使用帧级匈牙利匹配显著减少了重复和遗漏预测。
• 定性结果表明，采用帧级匹配的模型生成的超图更准确且时间上更一致，尤其在多步动作与关系上表现更优。
• 计算成本保持较低水平，仅使用 12 层 Transformer（编码器/解码器各 5 层），与现有视觉-语言模型（如 ALBEF）相当。

Figure 2 : The SHG-VQA architecture: we start with encoding the input video into spatio-temporal features using a pre-trained backbone. These video features are flattened into a sequence of tokens of length $T\times h\times w$ and position encoded to be further processed through a visual encoder. Th