[Paper Review] NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions

Junbin Xiao, Xindi Shang+2 more|arXiv (Cornell University)|May 18, 2021

Multimodal Machine Learning Applications63 references10 citations

TL;DR

NExT-QA introduces a large-scale, manually curated video question-answering benchmark with 5,440 videos and 52K question-answer pairs to advance video understanding beyond description toward causal and temporal action reasoning. Despite strong performance on descriptive questions, state-of-the-art models show significant failure in causal and temporal reasoning, especially in open-ended QA, revealing a fundamental gap in model understanding beyond answer selection.

ABSTRACT

We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions. Based on the dataset, we set up multi-choice and open-ended QA tasks targeting causal action reasoning, temporal action reasoning, and common scene comprehension. Through extensive analysis of baselines and established VideoQA techniques, we find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning. Furthermore, the models that are effective on multi-choice QA, when adapted to open-ended QA, still struggle in generalizing the answers. This raises doubt on the ability of these models to reason and highlights possibilities for improvement. With detailed results for different question types and heuristic observations for future works, we hope NExT-QA will guide the next generation of VQA research to go beyond superficial scene description towards a deeper understanding of videos. (The dataset and related resources are available at https://github.com/doc-doc/NExT-QA.git)

Motivation & Objective

• To advance video understanding from shallow description to deeper explanation of causal and temporal actions in videos.
• To address the lack of benchmarks that rigorously evaluate causal and temporal reasoning in real-world video content.
• To challenge existing VideoQA models by introducing multi-choice and open-ended QA tasks targeting reasoning beyond surface-level recognition.
• To provide a diagnostic tool for evaluating model generalization and reasoning capabilities in complex video understanding.
• To guide future research toward building models that truly understand action sequences and their underlying causes.

Proposed method

• The benchmark is constructed from 5,440 real-world videos sourced from VidOR, with manually annotated question-answer pairs grouped into causal, temporal, and descriptive categories.
• Two task types are defined: multi-choice QA with five candidate answers and open-ended QA requiring free-form answer generation from video and question cues.
• A comprehensive evaluation is conducted on state-of-the-art VideoQA models, including HGA, EVQA, PSAC+, and STVQA, across both task types.
• Performance is measured using accuracy for multi-choice QA and WUPS score for open-ended QA to assess semantic similarity of generated answers.
• Visual features are extracted using appearance and motion encoders, while text representations use BERT and GloVe for language modeling.
• Graph-based models (e.g., HGA) are explored to explicitly model temporal and causal relationships between actions and objects.

Experimental results

Research questions

• RQ1 Can state-of-the-art VideoQA models generalize beyond descriptive questions to reason about causal and temporal relationships in video actions?
• RQ2 How do multi-choice and open-ended QA tasks differ in exposing model limitations for reasoning and language generation?
• RQ3 To what extent do pre-trained language models like BERT improve reasoning on causal and temporal questions compared to word embeddings like GloVe?
• RQ4 Why do models that perform well on multi-choice QA fail when adapted to open-ended QA, and what does this imply about their reasoning capability?
• RQ5 How do visual feature fusion strategies (appearance vs. motion) impact performance on causal and temporal reasoning tasks?

Key findings

• Top-performing VideoQA models achieve strong results on descriptive questions (e.g., object and action recognition) but show a 10% performance gap on causal and temporal multi-choice questions.
• On open-ended QA, the performance gap widens to approximately 30% for causal and temporal questions, indicating severe limitations in reasoning and generation.
• Graph-based models like HGA show superior performance on causal and temporal reasoning, suggesting their value for modeling relational structures in videos.
• Pre-trained BERT representations outperform GloVe in reasoning tasks, especially for causal and temporal questions, despite BERT's bias toward textual QA.
• Models trained on multi-choice QA struggle significantly in open-ended QA, indicating they may rely on answer selection rather than grounded reasoning or language generation.
• The failure to generate coherent and accurate answers for longer questions and answers highlights weaknesses in language understanding and generation, even when visual features are available.