[Paper Review] Neural GPUs Learn Algorithms

ukasz Kaiser, Ilya Sutskever|arXiv (Cornell University)|Jan 1, 2016

Neural Networks and Applications63 citations

TL;DR

The paper introduces the Neural GPU, a highly parallel, differentiable recurrent architecture based on convolutional gated recurrent units that achieves computational universality and generalizes to long algorithmic tasks. It successfully learns binary addition and multiplication on sequences up to 20 bits during training and generalizes perfectly to much longer inputs, enabled by parameter sharing relaxation, dropout, and gradient noise.

ABSTRACT

Abstract: Learning an algorithm from examples is a fundamental problem that has been widely studied. Recently it has been addressed using neural networks, in particular by Neural Turing Machines (NTMs). These are fully differentiable computers that use backpropagation to learn their own programming. Despite their appeal NTMs have a weakness that is caused by their sequential nature: they are not parallel and are are hard to train due to their large depth when unfolded. We present a neural network architecture to address this problem: the Neural GPU. It is based on a type of convolutional gated recurrent unit and, like the NTM, is computationally universal. Unlike the NTM, the Neural GPU is highly parallel which makes it easier to train and efficient to run. An essential property of algorithms is their ability to handle inputs of arbitrary size. We show that the Neural GPU can be trained on short instances of an algorithmic task and successfully generalize to long instances. We verified it on a number of tasks including long addition and long multiplication of numbers represented in binary. We train the Neural GPU on numbers with upto 20 bits and observe no errors whatsoever while testing it, even on much longer numbers. To achieve these results we introduce a technique for training deep recurrent networks: parameter sharing relaxation. We also found a small amount of dropout and gradient noise to have a large positive effect on learning and generalization.

Motivation & Objective

• To address the sequential bottleneck and poor trainability of Neural Turing Machines (NTMs) by designing a more parallel, scalable architecture.
• To enable neural networks to learn and generalize algorithmic tasks on inputs of arbitrary length, including long binary arithmetic.
• To develop training techniques that stabilize deep recurrent networks and improve generalization beyond training sequence lengths.
• To demonstrate computational universality in a differentiable, parallel architecture suitable for end-to-end learning of algorithms.

Proposed method

• The Neural GPU employs a convolutional gated recurrent unit (Gated Recurrent Unit) with shared weights across spatial positions to enable parallel computation and reduce parameter count.
• It uses parameter sharing relaxation, a technique that decouples weight sharing during backpropagation to stabilize training of deep recurrent networks.
• The architecture is trained using backpropagation through time with gradient noise and dropout applied to improve generalization and training stability.
• The model processes sequences of binary digits and learns to perform operations like addition and multiplication by attending to relevant positions through its gated mechanism.
• The network is trained on short sequences (up to 20 bits) and evaluated on much longer sequences to test generalization.

Experimental results

Research questions

• RQ1 Can a differentiable, recurrent neural network architecture learn and generalize algorithmic tasks beyond the length seen during training?
• RQ2 How can deep recurrent networks be trained effectively when standard backpropagation fails due to vanishing or exploding gradients?
• RQ3 Can a parallel architecture achieve computational universality while maintaining the ability to learn complex algorithms?
• RQ4 What training techniques are most effective in improving generalization and stability in deep recurrent networks for algorithmic learning?

Key findings

• The Neural GPU achieved perfect generalization on binary addition and multiplication tasks, with no errors observed on sequences significantly longer than the 20-bit training sequences.
• The model demonstrated strong generalization to inputs of arbitrary length, indicating it had learned the underlying algorithmic structure rather than memorizing training examples.
• Parameter sharing relaxation significantly improved training stability and performance in deep recurrent architectures.
• The addition of small amounts of dropout and gradient noise substantially enhanced learning and generalization, suggesting regularization is critical for algorithmic learning.