[Paper Review] Learning Situation Hyper-Graphs for Video Question Answering

Aisha Urooj Khan, Hilde Kuehne+6 more|arXiv (Cornell University)|Apr 18, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes SHG-VQA, a novel end-to-end video question answering framework that learns situation hyper-graphs—representing actors, objects, relationships, and actions—from video clips without explicit object detection. By using a situation hyper-graph decoder to predict graph tokens and attending to them via cross-attention with question embeddings, the model achieves state-of-the-art performance on AGQA and STAR benchmarks, demonstrating that implicit hyper-graph learning significantly improves reasoning over complex video content.

ABSTRACT

Answering questions about complex situations in videos requires not only capturing the presence of actors, objects, and their relations but also the evolution of these relationships over time. A situation hyper-graph is a representation that describes situations as scene sub-graphs for video frames and hyper-edges for connected sub-graphs and has been proposed to capture all such information in a compact structured form. In this work, we propose an architecture for Video Question Answering (VQA) that enables answering questions related to video content by predicting situation hyper-graphs, coined Situation Hyper-Graph based Video Question Answering (SHG-VQA). To this end, we train a situation hyper-graph decoder to implicitly identify graph representations with actions and object/human-object relationships from the input video clip. and to use cross-attention between the predicted situation hyper-graphs and the question embedding to predict the correct answer. The proposed method is trained in an end-to-end manner and optimized by a VQA loss with the cross-entropy function and a Hungarian matching loss for the situation graph prediction. The effectiveness of the proposed architecture is extensively evaluated on two challenging benchmarks: AGQA and STAR. Our results show that learning the underlying situation hyper-graphs helps the system to significantly improve its performance for novel challenges of video question-answering tasks.

Motivation & Objective

• To address the challenge of video question answering in complex, dynamic scenarios requiring spatio-temporal reasoning.
• To improve reasoning in VQA by modeling situations as hyper-graphs that encode entities, relationships, and evolving actions over time.
• To learn structured scene representations implicitly from video input without relying on pre-trained object detectors or explicit graph computation.
• To optimize the model jointly for both accurate situation hyper-graph prediction and correct answer generation using a multi-task loss.
• To evaluate the effectiveness of hyper-graph learning on challenging benchmarks with dense ground-truth hyper-graph annotations.

Proposed method

• A situation hyper-graph decoder is introduced to predict atomic actions and object/actor-object relationships as a set of tokens from video clip features.
• The model uses cross-attention between the predicted hyper-graph tokens and the question embedding to generate answers.
• A bipartite matching loss (Hungarian matching) is applied to align predicted and ground-truth hyper-graph tokens at each frame, improving set prediction accuracy.
• The overall training objective combines a VQA loss (cross-entropy) and a Hungarian matching loss for action and relationship prediction.
• The architecture is end-to-end trainable and does not require explicit graph neural network layers during inference.
• The method treats spatio-temporal graph learning as a set prediction task, enabling direct prediction of relationships and actions without intermediate graph construction.

Figure 1 : The situation hyper-graph for a video is composed of situations with entities and their relationships (shown as subgraphs in the pink box). These situations may evolve over time. Temporal actions act as hyper-edges connecting these situations into one situation hyper-graph. Learning situa

Experimental results

Research questions

• RQ1 Can implicit learning of situation hyper-graphs from raw video improve video question answering performance?
• RQ2 Does modeling actions as hyper-edges connecting subgraphs enhance reasoning over dynamic video content?
• RQ3 How does frame-level Hungarian matching for hyper-graph prediction compare to full-video matching in terms of accuracy and temporal consistency?
• RQ4 To what extent does learning structured hyper-graph representations improve generalization on out-of-distribution reasoning tasks?
• RQ5 Can a transformer-based architecture achieve competitive performance in VQA without explicit graph computation or pre-defined scene graph heads?

Key findings

• The proposed SHG-VQA model achieves state-of-the-art performance on the AGQA benchmark, with a 39.90% accuracy using 4 actions and 12 relationships.
• On the STAR benchmark, the model reaches 39.20% accuracy with 3 actions and 8 relationships, demonstrating strong performance on complex reasoning types.
• Ablation studies show that high-quality hyper-graph prediction is critical for VQA performance, as models with poor graph generation fail to answer complex questions.
• The use of frame-level Hungarian matching for hyper-graph prediction significantly reduces duplicate and missing predictions compared to full-video matching.
• Qualitative results confirm that the model with frame-level matching produces more accurate and temporally consistent hyper-graphs, especially for multi-step actions and relationships.
• The computational cost remains low, with only 12 transformer layers (5 for each encoder/decoder), making it comparable to existing vision-language models like ALBEF.

Figure 2 : The SHG-VQA architecture: we start with encoding the input video into spatio-temporal features using a pre-trained backbone. These video features are flattened into a sequence of tokens of length $T\times h\times w$ and position encoded to be further processed through a visual encoder. Th