[Paper Review] Learning the Semantics of Manipulation Action

Yezhou Yang, Yiannis Aloimonos+2 more|arXiv (Cornell University)|Dec 4, 2015

Natural Language Processing Techniques3 citations

TL;DR

This paper proposes a Combinatory Categorial Grammar (CCG)-based framework that learns the semantic meaning of manipulation actions from annotated video data using probabilistic semantic parsing and lambda-calculus representations. The system enables robots to understand action semantics, reason about hidden consequences via propositional logic, and generalize beyond observation, achieving improved performance in action goal inference on the MANIAC dataset.

ABSTRACT

In this paper we present a formal computational framework for modeling manipulation actions. The introduced formalism leads to semantics of manipulation action and has applications to both observing and understanding human manipulation actions as well as executing them with a robotic mechanism (e.g. a humanoid robot). It is based on a Combinatory Categorial Grammar. The goal of the introduced framework is to: (1) represent manipulation actions with both syntax and semantic parts, where the semantic part employs $λ$-calculus; (2) enable a probabilistic semantic parsing schema to learn the $λ$-calculus representation of manipulation action from an annotated action corpus of videos; (3) use (1) and (2) to develop a system that visually observes manipulation actions and understands their meaning while it can reason beyond observations using propositional logic and axiom schemata. The experiments conducted on a public available large manipulation action dataset validate the theoretical framework and our implementation.

Motivation & Objective

• To develop a formal computational framework for modeling manipulation actions with both syntactic and semantic components.
• To learn action semantics from annotated video corpora using probabilistic semantic parsing.
• To enable reasoning beyond visual observation by encoding actions in formal logic for goal inference.
• To bridge the gap between action observation and execution in robotics through a unified semantic representation.

Proposed method

• The framework uses Combinatory Categorial Grammar (CCG) to represent actions with syntactic categories and semantic lambda-calculus expressions.
• It employs probabilistic semantic parsing to infer lambda-calculus representations of actions from annotated video data.
• Semantic representations are constructed using functional application and type-raising combinators to model action arguments and consequences.
• The system encodes action semantics using primitive functions (e.g., 'cut') and logical consequences (e.g., 'divided') in a formal logic form.
• Common sense axioms and propositional logic are used to infer hidden action outcomes and plan sequences of actions.
• The framework supports extension to complex actions via quantifiers (e.g., 'every'), temporal ordering, and additional combinatory rules.

Experimental results

Research questions

• RQ1 Can a formal grammar framework learn the semantic meaning of manipulation actions from video annotations?
• RQ2 How can learned semantic representations enable reasoning about unobserved action consequences?
• RQ3 Can the system infer action goals from visual observations using logical inference?
• RQ4 To what extent can the framework generalize to complex actions involving quantifiers and temporal ordering?
• RQ5 Can the framework support both action understanding and execution in a unified semantic representation?

Key findings

• The system successfully learned lambda-calculus representations of manipulation actions such as 'Cut' from a large annotated video dataset.
• The learned semantic representations enabled logical inference of hidden consequences, improving performance in action goal prediction.
• The framework demonstrated the ability to reason beyond observation using propositional logic and common sense axioms.
• The system achieved promising results on the public MANIAC dataset, validating the theoretical framework.
• Extension to quantified expressions (e.g., 'every tomato') and temporal ordering was shown to be feasible within the CCG framework.
• The approach enables robots to imitate not just actions, but also action goals, by reasoning about semantic consequences.