[Paper Review] Deeply Semantic Inductive Spatio-Temporal Learning

Jakob Suchan, Mehul Bhatt+1 more|arXiv (Cornell University)|Aug 9, 2016

Constraint Satisfaction and Optimization20 references3 citations

TL;DR

This paper proposes a novel inductive logic programming (ILP) framework for learning relational spatio-temporal structures from dynamic visuo-spatial data, natively integrating qualitative and quantitative spatial semantics. The system learns perceptual axioms from eye-tracking and video data, enabling explainable, declarative reasoning about symmetry and attention dynamics in cinematic and cognitive science applications.

ABSTRACT

We present an inductive spatio-temporal learning framework rooted in inductive logic programming. With an emphasis on visuo-spatial language, logic, and cognition, the framework supports learning with relational spatio-temporal features identifiable in a range of domains involving the processing and interpretation of dynamic visuo-spatial imagery. We present a prototypical system, and an example application in the domain of computing for visual arts and computational cognitive science.

Motivation & Objective

• To develop a systematic inductive logic programming framework for learning commonsense knowledge grounded in spatial, temporal, and spatio-temporal relations from dynamic visuo-spatial imagery.
• To enable seamless integration of qualitative and quantitative spatial data within a logic programming environment for cognitive and perceptual reasoning.
• To support the learning of declarative, semantically grounded spatio-temporal axioms from real-world data such as video, point clouds, and eye-tracking.
• To provide a unified ontology for relational spatio-temporal features (e.g., topology, orientation, distance, size) directly embedded in the ILP learning process.
• To demonstrate applicability in cognitive science and visual arts by learning perceptual patterns like symmetry and attention switching from cinematic scenes.

Proposed method

• The framework employs a formal spatio-temporal ontology $\mathcal{O}_{sp} = \langle \mathcal{E}, \mathcal{R} \rangle$, where $\mathcal{E}$ defines basic spatial entities (points, vectors, rectangles, line segments) and $\mathcal{R}$ defines relational features (topology, orientation, distance, size).
• Spatial relations are natively encoded in the ILP system, ensuring built-in semantics for qualitative and quantitative reasoning without external mapping.
• The system uses the predicate $\mathsf{holds\text{-}in/2}$ to represent spatio-temporal relations at specific time points, enabling modeling of dynamic changes such as attention switches.
• Learning is performed via ILP over relational facts derived from computer vision outputs (e.g., bounding boxes, face directions) and eye-tracking data.
• A prototype system is implemented within a constraint logic programming framework, enabling integration with external knowledge sources and reasoning modules.
• The framework supports inductive-abductive inference, allowing for hypothesis formation and explanation of perceptual patterns in dynamic scenes.

Experimental results

Research questions

• RQ1 How can a learning framework natively support the semantics of complex spatio-temporal relations (e.g., symmetry, attention shifts) without relying on external semantic mappings?
• RQ2 To what extent can ILP be extended to learn qualitative spatial structures—beyond topological relations—such as orientation, distance, and size—from visual data?
• RQ3 Can perceptual axioms, such as those governing attention switching in cinematic scenes, be learned from dynamic eye-tracking and motion data using a declarative logic-based approach?
• RQ4 How can mixed qualitative and quantitative spatial data be uniformly represented and reasoned about within a single inductive learning framework?
• RQ5 What is the feasibility of using such a system for explainable, cognition-grounded learning in visual perception and artistic analysis?

Key findings

• The framework successfully learns relational spatial structures such as object-level symmetry in cinematic scenes, using bounding boxes and face directions as input features.
• The system learns axioms of visual perception from eye-tracking data, modeling attention switches between individuals as dynamic spatio-temporal relations.
• The integration of qualitative and quantitative spatial features (e.g., distance, orientation, size) is natively supported within the ILP framework, enabling mixed-reasoning capabilities.
• The prototype demonstrates functional viability in learning perceptual patterns from real-world data, including dynamic scenes from film and human behavioral tracking.
• The approach enables explainable, declarative reasoning about spatio-temporal dynamics, with results interpretable as logical rules grounded in spatial semantics.
• To the best of the authors’ knowledge, this is the first ILP-based framework with built-in semantics for a broad range of domain-independent spatio-temporal relations beyond topological ones.