[Paper Review] HySTER: A Hybrid Spatio-Temporal Event Reasoner

Théophile Sautory, Nuri Cingillioglu+1 more|arXiv (Cornell University)|Jan 17, 2021

Semantic Web and Ontologies28 references4 citations

TL;DR

HySTER is a neuro-symbolic VideoQA model that combines deep learning for visual perception with answer set programming (ASP) for symbolic reasoning over temporal, causal, and physical events. It achieves state-of-the-art accuracy on the CLEVRER dataset by leveraging domain-independent rules for event detection and causal reasoning, enabling explainable and generalizable video understanding.

ABSTRACT

The task of Video Question Answering (VideoQA) consists in answering natural language questions about a video and serves as a proxy to evaluate the performance of a model in scene sequence understanding. Most methods designed for VideoQA up-to-date are end-to-end deep learning architectures which struggle at complex temporal and causal reasoning and provide limited transparency in reasoning steps. We present the HySTER: a Hybrid Spatio-Temporal Event Reasoner to reason over physical events in videos. Our model leverages the strength of deep learning methods to extract information from video frames with the reasoning capabilities and explainability of symbolic artificial intelligence in an answer set programming framework. We define a method based on general temporal, causal and physics rules which can be transferred across tasks. We apply our model to the CLEVRER dataset and demonstrate state-of-the-art results in question answering accuracy. This work sets the foundations for the incorporation of inductive logic programming in the field of VideoQA.

Motivation & Objective

• To address the limitations of end-to-end deep learning models in VideoQA, which lack transparency and struggle with complex temporal and causal reasoning.
• To integrate symbolic AI, specifically answer set programming (ASP), into VideoQA to enable compositional, explainable reasoning over physical events in videos.
• To develop a generalizable framework that combines neural perception with symbolic rules for temporal, causal, and physical event reasoning.
• To demonstrate that neuro-symbolic integration improves performance on reasoning-intensive VideoQA benchmarks like CLEVRER.
• To lay the foundation for incorporating inductive logic programming (ILP) into VideoQA by enabling rule learning from data.

Proposed method

• The model uses Mask R-CNN as a perception module to extract object detections and features from video frames, which are then mapped into symbolic representations.
• A symbolic scene representation is constructed using facts in ASP, encoding object identities, positions, velocities, and physical properties at each time step.
• The system employs the Event Calculus framework to model temporal and causal relationships, using predicates such as $happens(E,T)$, $holdsAt(F,T)$, $initiates(E,F)$, and $terminates(E,F)$.
• Domain-independent rules for physics, causality, and temporal logic are encoded in ASP to reason about events such as collisions, velocity changes, and object interactions.
• Event detection rules are engineered using geometric distance and velocity change thresholds to identify physical events like collisions, with thresholds optimized on training data.
• The reasoning pipeline combines object-level symbolic facts with logical rules to answer natural language questions via query evaluation over answer sets.

Experimental results

Research questions

• RQ1 Can a neuro-symbolic approach combining deep learning and symbolic reasoning outperform end-to-end deep learning models in VideoQA tasks requiring complex temporal and causal reasoning?
• RQ2 How effectively can answer set programming (ASP) model and reason about physical events such as collisions in videos using general-purpose temporal and causal rules?
• RQ3 To what extent can engineered symbolic rules for event detection improve accuracy on reasoning-intensive VideoQA benchmarks like CLEVRER?
• RQ4 Can symbolic reasoning with ASP handle compositional questions involving negation and counting, which are challenging for pure neural networks?
• RQ5 Can the framework be generalized to support automatic rule learning via inductive logic programming (ILP), reducing reliance on handcrafted rules?

Key findings

• HySTER achieves state-of-the-art performance on the CLEVRER dataset, outperforming existing models in question-answering accuracy.
• The perception module using Mask R-CNN achieves an F1 score of 0.991 on object detection and classification across 1,000 training videos, ensuring high-fidelity symbolic input.
• The velocity change rule effectively distinguishes colliding from non-colliding object pairs, with distributions showing clear separation before and during collisions.
• The Euclidean distance threshold for collision detection successfully filters out non-colliding pairs while preserving most actual collisions, as shown in Figure 2.
• The symbolic reasoning component enables accurate handling of compositional questions involving negation and counting, which remain challenging for purely neural models.
• The model demonstrates robustness in detecting events that span multiple time frames, as evidenced by consistent velocity change patterns across frames before, during, and after collisions.