[论文解读] Neural GPUs Learn Algorithms

ukasz Kaiser, Ilya Sutskever|arXiv (Cornell University)|Jan 1, 2016

Neural Networks and Applications被引用 63

一句话总结

该论文提出了神经GPU（Neural GPU），一种高度并行、可微分的循环架构，基于卷积门控循环单元，实现了计算通用性，并能泛化到长算法任务。该模型在训练期间成功学习了长度达20位的二进制加法和乘法，并能完美泛化到远超训练长度的输入，这得益于参数共享松弛、dropout和梯度噪声的使用。

ABSTRACT

Abstract: Learning an algorithm from examples is a fundamental problem that has been widely studied. Recently it has been addressed using neural networks, in particular by Neural Turing Machines (NTMs). These are fully differentiable computers that use backpropagation to learn their own programming. Despite their appeal NTMs have a weakness that is caused by their sequential nature: they are not parallel and are are hard to train due to their large depth when unfolded. We present a neural network architecture to address this problem: the Neural GPU. It is based on a type of convolutional gated recurrent unit and, like the NTM, is computationally universal. Unlike the NTM, the Neural GPU is highly parallel which makes it easier to train and efficient to run. An essential property of algorithms is their ability to handle inputs of arbitrary size. We show that the Neural GPU can be trained on short instances of an algorithmic task and successfully generalize to long instances. We verified it on a number of tasks including long addition and long multiplication of numbers represented in binary. We train the Neural GPU on numbers with upto 20 bits and observe no errors whatsoever while testing it, even on much longer numbers. To achieve these results we introduce a technique for training deep recurrent networks: parameter sharing relaxation. We also found a small amount of dropout and gradient noise to have a large positive effect on learning and generalization.

研究动机与目标

• 通过设计一种更并行、可扩展的架构，解决神经图灵机（NTM）的序列瓶颈和训练困难问题。
• 使神经网络能够学习并泛化任意长度输入的算法任务，包括长二进制算术运算。
• 开发能够稳定深层循环网络并提升其在训练序列长度之外泛化能力的训练技术。
• 在一种可微分、并行的架构中展示计算通用性，使其适用于算法端到端学习。

提出的方法

• 神经GPU采用在空间位置间共享权重的卷积门控循环单元（Gated Recurrent Unit），以实现并行计算并减少参数数量。
• 采用参数共享松弛技术，该技术在反向传播过程中解耦权重共享，以稳定深层循环网络的训练。
• 使用时间反向传播进行训练，并应用梯度噪声和dropout以提升泛化能力和训练稳定性。
• 网络处理二进制数字序列，并通过其门控机制关注相关位置，学习执行加法和乘法等操作。
• 模型在短序列（最多20位）上进行训练，并在远超训练长度的序列上进行评估，以测试泛化能力。

实验结果

研究问题

• RQ1 可微分的循环神经网络架构是否能够学习并泛化超出训练时所见长度的算法任务？
• RQ2 当标准反向传播因梯度消失或爆炸而失效时，如何有效训练深层循环网络？
• RQ3 并行架构是否能够在保持学习复杂算法能力的同时实现计算通用性？
• RQ4 在深层循环网络中，哪些训练技术最有效地提升算法学习的泛化能力和稳定性？

主要发现

• 神经GPU在二进制加法和乘法任务中实现了完美泛化，在远超20位的测试序列中未观察到任何错误。
• 该模型在任意长度输入上表现出强大的泛化能力，表明其已学习到底层算法结构，而非记忆训练样本。
• 参数共享松弛显著提升了深层循环架构的训练稳定性和性能。
• 引入少量dropout和梯度噪声显著增强了学习和泛化能力，表明正则化对算法学习至关重要。