[Paper Review] A disembodied developmental robotic agent called Samu Bátfai

Norbert Bátfai|arXiv (Cornell University)|Nov 9, 2015

Modular Robots and Swarm Intelligence3 citations

TL;DR

This paper presents Samu Bátfai, a disembodied developmental robotic agent that uses deep Q-learning with neural network function approximators to learn natural language conversation through reinforcement learning. By predicting the next sentence in tales or dialogues using SPO (Subject-Predicate-Object) triplets and leveraging LZW tree compression to reduce action space, Samu achieves significant learning improvement, demonstrating a viable prototype for developmental robotics using only text-based interaction.

ABSTRACT

The agent program, called Samu, is an experiment to build a disembodied DevRob (Developmental Robotics) chatter bot that can talk in a natural language like humans do. One of the main design feature is that Samu can be interacted with using only a character terminal. This is important not only for practical aspects of Turing test or Loebner prize, but also for the study of basic principles of Developmental Robotics. Our purpose is to create a rapid prototype of Q-learning with neural network approximators for Samu. We sketch out the early stages of the development process of this prototype, where Samu's task is to predict the next sentence of tales or conversations. The basic objective of this paper is to reach the same results using reinforcement learning with general function approximators that can be achieved by using the classical Q lookup table on small input samples. The paper is closed by an experiment that shows a significant improvement in Samu's learning when using LZW tree to narrow the number of possible Q-actions.

Motivation & Objective

• To develop a disembodied, chat-based developmental robotic agent capable of learning natural language understanding and generation through reinforcement learning.
• To implement a rapid prototype of deep Q-learning with neural network function approximators for language prediction in a constrained, text-only environment.
• To investigate how function approximation and action space reduction techniques can improve sample efficiency in developmental language learning.
• To explore the feasibility of training a language agent using only verbal input, without visual or auditory modalities, to study core principles of developmental robotics.
• To create a scalable, low-resource system that mimics early childhood language acquisition through reinforcement learning on natural language corpora.

Proposed method

• Samu uses a simplified architecture where the Q-learning agent predicts the next sentence in a conversation or story based on SPO (Subject-Predicate-Object) triplets derived from input text.
• The agent employs multilayer perceptrons as function approximators to estimate the Q-value function, enabling deep reinforcement learning for sequence prediction.
• The action space is reduced using an LZW tree to compress and narrow the number of possible next sentence actions, improving learning efficiency.
• A SARSA-based Q-learning algorithm is applied, with rewards based on matching predicted sentences to actual input sentences, enabling online learning from sequential dialogue or narrative data.
• The system uses a frequency table to index state-action pairs, with state represented by current SPO triplet or sequence context, and updates Q-values using temporal difference learning.
• The agent operates solely through a character terminal, with no visual or auditory input, enforcing a purely linguistic developmental learning path.

Experimental results

Research questions

• RQ1 Can a developmental robotic agent learn to predict natural language sequences using only text-based interaction and reinforcement learning?
• RQ2 How effective is deep Q-learning with neural network function approximators in learning language generation without explicit supervision?
• RQ3 To what extent does action space reduction via LZW tree compression improve learning speed and accuracy in a language prediction task?
• RQ4 Can a purely verbal, disembodied agent simulate early-stage language development akin to a 2–4 year old child?
• RQ5 How does the use of SPO triplets as a representation for linguistic meaning support generalization and learning in a reinforcement learning framework?

Key findings

• The use of LZW tree compression significantly improves Samu’s learning performance by reducing the number of possible Q-actions, leading to faster convergence in the prediction task.
• Samu successfully implements deep Q-learning with neural network approximators for natural language sequence prediction, achieving results comparable to classical Q-learning with lookup tables on small datasets.
• The system demonstrates that a purely text-based, disembodied agent can learn to predict sentence continuations through reinforcement learning, even without visual or auditory input.
• The memory footprint of the agent is manageable in practice due to the Pareto-like distribution of SPO triplets, reducing the effective number of required perceptrons.
• The prototype can be reproduced on a standard PC, indicating low computational requirements despite the complexity of the underlying function approximation.
• The agent's behavior is designed to align with developmental robotics principles, including a prenatal learning phase and caregiver-mediated knowledge verification, simulating early childhood language acquisition.