[Paper Review] Action Categorization for Computationally Improved Task Learning and Planning

Lakshmi Nair, Sonia Chernova|arXiv (Cornell University)|Apr 26, 2018

Reinforcement Learning in Robotics26 references3 citations

TL;DR

This paper introduces the Action-Category Representation (ACR), an algorithm-agnostic, abstract data structure that maps objects to pre-defined action categories using action codes—inspired by human cognitive processes. ACR reduces the action space in planning and reinforcement learning, leading to significant computational improvements and faster learning, especially with limited or imperfect demonstrations.

ABSTRACT

This paper explores the problem of task learning and planning, contributing the Action-Category Representation (ACR) to improve computational performance of both Planning and Reinforcement Learning (RL). ACR is an algorithm-agnostic, abstract data representation that maps objects to action categories (groups of actions), inspired by the psychological concept of action codes. We validate our approach in StarCraft and Lightworld domains; our results demonstrate several benefits of ACR relating to improved computational performance of planning and RL, by reducing the action space for the agent.

Motivation & Objective

• To address the computational inefficiency of planning and reinforcement learning (RL) in large action spaces by introducing a cognitive-inspired abstraction mechanism.
• To reduce the number of actions the agent must consider during decision-making, thereby improving scalability and performance.
• To enable efficient learning from few or suboptimal demonstrations by constraining action selection via learned object-action categories.
• To develop a representation that is both algorithm-agnostic and extensible to continuous domains through state-space discretization.
• To combine ACR with existing methods like HAT to further improve sample efficiency and robustness to demonstration errors.

Proposed method

• ACR uses action codes—tuples of object sets and associated actions—to group semantically related actions (e.g., ((apple,knife),(cut,peel))—a cognitive-inspired abstraction.
• The representation is constructed from agent experience or human demonstrations, enabling online learning of new object-action categories.
• ACR restricts action selection during planning and RL by limiting actions to those in the action category associated with current objects.
• In PDDL planning, ACR reduces planning time by pruning irrelevant actions based on object-action mappings.
• In Q-learning, ACR improves learning speed and convergence by reducing the action space, especially with sparse demonstrations.
• ACR is combined with Human-Agent Transfer (HAT) by validating that HAT’s suggested actions are consistent with ACR’s action categories, enhancing early exploration and policy learning.

Experimental results

Research questions

• RQ1 Can an abstract action categorization mechanism reduce computational complexity in planning and reinforcement learning?
• RQ2 How does ACR perform when only a single or suboptimal demonstration is available for learning?
• RQ3 To what extent does ACR improve learning efficiency and planning speed compared to baseline methods?
• RQ4 Can ACR be effectively combined with existing imitation learning frameworks like HAT to enhance sample efficiency?
• RQ5 How does ACR generalize to unseen objects or new task contexts without requiring new demonstrations?

Key findings

• ACR reduces the action space by grouping semantically related actions, leading to faster planning and learning without sacrificing solution quality.
• In the StarCraft domain, ACR significantly reduced planning time by restricting action sets based on object-action categories.
• With only one expert demonstration, ACR outperformed baseline methods and HAT in learning performance, demonstrating robustness to limited data.
• Combining ACR with HAT improved learning performance beyond either method used alone, especially in early training phases.
• ACR achieved better learning convergence and reduced exploration time in Q-learning, even when demonstrations were suboptimal.
• The method showed strong scalability and adaptability, with formal computational bounds guaranteeing no worse performance than traditional methods.