[Paper Review] Reinforcement Learning Neural Turing Machines - Revised

Wojciech Zaremba, Ilya Sutskever|arXiv (Cornell University)|May 4, 2015

Reinforcement Learning in RoboticsComputer Science22 references118 citations

TL;DR

This paper proposes the Reinforcement Learning Neural Turing Machine (RL-NTM), which uses the REINFORCE algorithm to train a neural controller to interact with discrete external interfaces—input tape, memory tape, and output tape—enabling Turing-complete computation. The model successfully solves algorithmic tasks like sequence copying, reversal, and repeat-copy, but performance is highly dependent on controller architecture and requires careful gradient validation via a novel numerical gradient check procedure.

ABSTRACT

The Neural Turing Machine (NTM) is more expressive than all previously considered models because of its external memory. It can be viewed as a broader effort to use abstract external Interfaces and to learn a parametric model that interacts with them. The capabilities of a model can be extended by providing it with proper Interfaces that interact with the world. These external Interfaces include memory, a database, a search engine, or a piece of software such as a theorem verifier. Some of these Interfaces are provided by the developers of the model. However, many important existing Interfaces, such as databases and search engines, are discrete. We examine feasibility of learning models to interact with discrete Interfaces. We investigate the following discrete Interfaces: a memory Tape, an input Tape, and an output Tape. We use a Reinforcement Learning algorithm to train a neural network that interacts with such Interfaces to solve simple algorithmic tasks. Our Interfaces are expressive enough to make our model Turing complete.

Motivation & Objective

• To investigate the feasibility of training neural networks to interact with discrete, non-differentiable external interfaces such as memory tapes and input/output tapes.
• To extend the capabilities of neural models by enabling them to learn control policies for discrete interfaces using reinforcement learning, rather than backpropagation alone.
• To demonstrate that a model combining differentiable memory writing with discrete action selection (via REINFORCE) can achieve Turing completeness and solve complex algorithmic tasks.
• To develop and validate a robust gradient checking procedure for REINFORCE-based training in models with hybrid differentiable and discrete components.

Proposed method

• The RL-NTM architecture consists of a neural controller that uses REINFORCE to decide discrete actions: moving the input, memory, and output tape heads, and whether to make a prediction.
• The controller uses backpropagation to learn what values to write to the memory and output tapes, while REINFORCE trains the policy for discrete actions such as tape head movements and output decisions.
• The model is trained using stochastic gradient descent with fixed learning rate (0.05) and momentum (0.9), with gradient norms clipped to 5 for RL parameters and 2 for the baseline network.
• A numerical gradient checking procedure is developed and applied to validate the REINFORCE gradients, which is critical due to the complexity of interactions between differentiable and discrete components.
• The memory is initialized as a 35-dimensional vector, and all hidden states and memory are initialized to zero.
• The controller uses a Gaussian initialization (σ = 0.1) and an inverse temperature of 0.01 to stabilize policy gradient updates.

Experimental results

Research questions

• RQ1 Can reinforcement learning be effectively used to train a neural controller to interact with discrete external interfaces such as memory and input/output tapes?
• RQ2 Does combining differentiable weight updates with discrete policy learning (via REINFORCE) enable the model to solve complex algorithmic tasks like sequence reversal and repeat-copy?
• RQ3 How sensitive is the model’s performance to the architecture of the controller, particularly in comparison to direct access or LSTM-based controllers?
• RQ4 Can a gradient checking procedure be reliably applied to REINFORCE-based models with hybrid differentiable and discrete components?
• RQ5 To what extent does curriculum learning improve training success for tasks like repeat-copy and forward-reverse?

Key findings

• The RL-NTM successfully solves the repeat-copy, reverse, and forward-reverse tasks when using a direct access controller, but fails completely with an LSTM controller.
• The model achieves success on tasks requiring unbounded memory and output decisions, demonstrating Turing completeness in principle.
• Without curriculum learning, the model fails to solve tasks beyond very short sequences (e.g., length 5), indicating strong dependency on training schedule.
• The gradient checking procedure developed in this work proved essential for debugging and validating the REINFORCE updates, which were otherwise difficult to verify.
• The model fails to solve long integer addition and sorting tasks, often learning ad-hoc, memory-intensive heuristics rather than generalizable algorithms.
• The success of the model is highly sensitive to controller design, with direct access controllers outperforming LSTM-based ones on all tested algorithmic tasks.