[Paper Review] Towards Deep Symbolic Reinforcement Learning

Marta Garnelo, Kai Arulkumaran+1 more|arXiv (Cornell University)|Sep 18, 2016

Reinforcement Learning in Robotics29 references145 citations

TL;DR

This paper proposes a hybrid neural-symbolic reinforcement learning architecture with a neural back end for symbol grounding and a symbolic front end for policy learning, showing data-efficient learning and transfer advantages over fully neural DRL in simple game variants.

ABSTRACT

Deep reinforcement learning (DRL) brings the power of deep neural networks to bear on the generic task of trial-and-error learning, and its effectiveness has been convincingly demonstrated on tasks such as Atari video games and the game of Go. However, contemporary DRL systems inherit a number of shortcomings from the current generation of deep learning techniques. For example, they require very large datasets to work effectively, entailing that they are slow to learn even when such datasets are available. Moreover, they lack the ability to reason on an abstract level, which makes it difficult to implement high-level cognitive functions such as transfer learning, analogical reasoning, and hypothesis-based reasoning. Finally, their operation is largely opaque to humans, rendering them unsuitable for domains in which verifiability is important. In this paper, we propose an end-to-end reinforcement learning architecture comprising a neural back end and a symbolic front end with the potential to overcome each of these shortcomings. As proof-of-concept, we present a preliminary implementation of the architecture and apply it to several variants of a simple video game. We show that the resulting system -- though just a prototype -- learns effectively, and, by acquiring a set of symbolic rules that are easily comprehensible to humans, dramatically outperforms a conventional, fully neural DRL system on a stochastic variant of the game.

Motivation & Objective

• Motivate and address key DRL shortcomings: data inefficiency, brittleness, lack of high-level reasoning, and opacity.
• Propose an end-to-end architecture combining a neural back end for symbol grounding with a symbolic front end for decision making.
• Demonstrate a proof-of-concept implementation on simple video-game variants to illustrate benefits of symbolic reasoning.
• Highlight potential for transfer learning and transparency through symbolic representations.

Proposed method

• Three-stage pipeline: low-level symbol generation via a convolutional autoencoder to produce symbolic tokens.
• Representation building that tracks objects over time using object persistence, types, and relations to form a spatio-temporal symbolic state.
• Reinforcement learning with a localized, compositional approach that trains separate Q-functions for interactions between object types and combines them for action selection.

Experimental results

Research questions

• RQ1 Can a neural back end learn a compositional, grounded symbolic representation from raw perceptual data?
• RQ2 Does a symbolic front end enable data-efficient learning and transfer in reinforcement learning tasks?
• RQ3 What are the benefits of local, object-type–coupled Q-functions versus a monolithic neural policy in simple environments?
• RQ4 How does the proposed architecture compare to conventional DRL (DQN) on variants of a simple game with differing object types and randomization?

Key findings

• The hybrid architecture learns effectively on all four game variants in the prototype setup.
• In the most challenging random-object variant, the symbolic approach dramatically outperforms DQN, learning a competent policy where DQN fails within 1000 epochs.
• The system achieves transfer-like benefits, generalizing to new variants without re-training the back end.
• The symbolic front end provides a human-understandable chain of justification for actions via the involved Q-functions and object interactions.