[Paper Review] Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition

Hao Fei, Shengqiong Wu+5 more|arXiv (Cornell University)|May 7, 2024

Aesthetic Perception and AnalysisNeuroscience3 citations

TL;DR

This paper introduces Video-of-Thought (VoT), a novel reasoning framework that enables human-level video understanding by decomposing complex video tasks into step-by-step perceptual and cognitive sub-problems. Built on MotionEpic, a video Multimodal Large Language Model (MLLM) with fine-grained spatial-temporal scene graph (STSG) grounding, VoT achieves state-of-the-art performance on complex video QA benchmarks by integrating pixel-level perception, trajectory tracking, and commonsense reasoning in a chain-of-thought manner.

ABSTRACT

Existing research of video understanding still struggles to achieve in-depth comprehension and reasoning in complex videos, primarily due to the under-exploration of two key bottlenecks: fine-grained spatial-temporal perceptive understanding and cognitive-level video scene comprehension. This paper bridges the gap by presenting a novel solution. We first introduce a novel video Multimodal Large Language Model (MLLM), MotionEpic, which achieves fine-grained pixel-level spatial-temporal video grounding by integrating video spatial-temporal scene graph (STSG) representation. Building upon MotionEpic, we then develop a Video-of-Thought (VoT) reasoning framework. VoT inherits the Chain-of-Thought (CoT) core, breaking down a complex task into simpler and manageable sub-problems, and addressing them step-by-step from a low-level pixel perception to high-level cognitive interpretation. Extensive experiments across various complex video QA benchmarks demonstrate that our overall framework strikingly boosts existing state-of-the-art. To our knowledge, this is the first attempt at successfully implementing the CoT technique for achieving human-level video reasoning, where we show great potential in extending it to a wider range of video understanding scenarios. Project is open at https://haofei.vip/VoT

Motivation & Objective

• To address the gap in deep video understanding, particularly in complex videos requiring fine-grained spatiotemporal perception and cognitive-level semantics.
• To overcome the limitations of existing video MLLMs that focus on shallow perceptual understanding and lack structured reasoning.
• To pioneer the integration of Chain-of-Thought (CoT) prompting with video inputs, enabling multi-hop reasoning from pixel-level grounding to high-level cognitive interpretation.
• To develop a video MLLM, MotionEpic, capable of grounding video content to STSG representations and supporting both STSG-aware and STSG-free inference.
• To demonstrate that structured, step-by-step reasoning significantly improves accuracy and robustness in complex video question answering.

Proposed method

• Proposes MotionEpic, a video MLLM that jointly encodes video frames and spatial-temporal scene graphs (STSGs) for fine-grained pixel-level grounding.
• Introduces a grounding-aware tuning phase using STSG annotations to align video inputs with structured scene representations.
• Employs a two-stage training strategy: first, grounding video to STSG via supervision; second, fine-tuning for autonomous STSG parsing and reasoning without STSG input.
• Designs the Video-of-Thought (VoT) framework as a CoT-style reasoning engine that decomposes video QA into sequential steps: target grounding, trajectory tracking, interaction analysis, and answer ranking.
• Leverages factual commonsense knowledge to interpret actions and interactions, ensuring cognitively coherent and explainable reasoning.
• Uses loss components 𝒫₂ and 𝒫₄ to explicitly optimize STSG grounding and generation, which are critical for downstream reasoning performance.

Figure 1: Human-like video reasoning intuitively follows a multi-step procedure, from lower-level perceptive fine-grained pixel grounding and tracking, to higher-level cognitive action scene semantics understanding.

Experimental results

Research questions

• RQ1 Can a video MLLM with fine-grained STSG grounding significantly improve perception accuracy in complex video understanding?
• RQ2 How effective is the Chain-of-Thought (CoT) paradigm when adapted to video reasoning with multimodal inputs?
• RQ3 To what extent does integrating commonsense knowledge enhance reasoning over video content beyond visual perception?
• RQ4 What role does structured scene representation (STSG) play in reducing hallucinations and improving reasoning robustness?
• RQ5 Can a unified framework support both STSG-aware and STSG-free inference while maintaining high performance?

Key findings

• MotionEpic with the VoT framework achieves state-of-the-art performance on complex video QA benchmarks, outperforming existing models including Video-LLaVA with CoT.
• The ablation study shows that loss components 𝒫₂ and 𝒫₄—related to STSG grounding and generation—cause the greatest performance degradation when removed, proving their critical role.
• Human evaluation on 200 hard examples shows that MotionEpic with VoT performs comparably to human-level reasoning, especially in action semantics and commonsense understanding.
• Qualitative case studies demonstrate that VoT correctly identifies complex scenes, such as a trainer training a dog, by combining accurate grounding with factual reasoning, while baselines fail.
• The VoT framework reduces error rates significantly compared to baseline CoT methods, particularly in reasoning about causal dynamics and social interactions.
• The system maintains strong zero-shot generalization ability, even when tested on unseen video reasoning tasks, due to its structured, step-by-step reasoning process.

Figure 2: Overview of the MotionEpic video MLLM.