[Paper Review] Deep Semantic Abstractions of Everyday Human Activities: On Commonsense Representations of Human Interactions

Jakob Suchan, Mehul Bhatt|arXiv (Cornell University)|Oct 10, 2017

Semantic Web and Ontologies3 citations

TL;DR

This paper proposes a deep semantic ontology for modeling human-object interactions in everyday activities using commonsense spatial and temporal reasoning grounded in RGB-D sensor data. It formalizes spatio-temporal relations through constraint logic programming, enabling declarative, explainable reasoning about motion and interaction patterns in cognitive robotics settings.

ABSTRACT

We propose a deep semantic characterization of space and motion categorically from the viewpoint of grounding embodied human-object interactions. Our key focus is on an ontological model that would be adept to formalisation from the viewpoint of commonsense knowledge representation, relational learning, and qualitative reasoning about space and motion in cognitive robotics settings. We demonstrate key aspects of the space & motion ontology and its formalization as a representational framework in the backdrop of select examples from a dataset of everyday activities. Furthermore, focussing on human-object interaction data obtained from RGBD sensors, we also illustrate how declarative (spatio-temporal) reasoning in the (constraint) logic programming family may be performed with the developed deep semantic abstractions.

Motivation & Objective

• To develop an ontological framework for representing embodied human-object interactions grounded in commonsense knowledge of space and motion.
• To formalize qualitative spatio-temporal relations (e.g., touching, during, part-of) for use in cognitive robotics and explainable AI.
• To enable declarative reasoning about dynamic visuo-spatial scenes using logic programming paradigms such as CLP(QS) and ASPMT(QS).
• To bridge high-level conceptual representations of actions with low-level sensory-motor data from RGB-D sensors.
• To support relational learning, abductive reasoning, and simulation grounded in cognitive linguistics and spatial cognition.

Proposed method

• Designs an ontological model of space and motion centered on commonsense representations of human-object interactions.
• Uses RGB-D sensor data to extract 3D joint positions and object locations over time, forming spatio-temporal tracks.
• Encodes interactions (e.g., pick, pass, pour) as sequences of qualitative spatio-temporal relations grounded in observed dynamics.
• Applies constraint logic programming (CLP(QS)) to perform declarative reasoning over space-time histories and motion patterns.
• Integrates the formal model with rule-based inference for non-monotonic reasoning, abductive explanation, and relational learning.
• Validates the framework on a dataset of everyday kitchen activities, using Prolog for interactive query answering on interaction sequences.

Experimental results

Research questions

• RQ1 How can commonsense spatial and temporal relations be formally modeled to support reasoning about human-object interactions in everyday activities?
• RQ2 In what way can deep semantic abstractions of space and motion be grounded in low-level RGB-D sensor data for cognitive robotics?
• RQ3 How can constraint logic programming be used to perform declarative, explainable reasoning about spatio-temporal dynamics in human-robot interaction?
• RQ4 What are the key qualitative relations and motion patterns that distinguish safe vs. unsafe interactions (e.g., passing a cup over a laptop)?
• RQ5 How can the ontology be extended to support indoor mobility and social interactions in more complex environments?

Key findings

• The proposed ontology successfully models human-object interactions using qualitative spatio-temporal relations such as 'touching', 'during', and 'part-of' in a formal, machine-processable way.
• The framework enables interactive query-based reasoning over interaction sequences, allowing detection of grounded actions like 'passing a cup' with spatial context.
• The system distinguishes safe from unsafe behaviors—e.g., passing an empty cup over a laptop versus a full one—based on spatial and temporal constraints.
• Constraint logic programming (CLP(QS)) effectively supports mixed quantitative-qualitative inference, enabling consistency checks and abductive explanations.
• The model supports relational learning and simulation grounded in cognitive linguistics, facilitating knowledge acquisition from natural language and sensor data.
• The approach is extensible to real-world robotic platforms, with integration planned into ROS and the openEASE and ExpCog cognition robotics frameworks.