[Paper Review] Learning to Execute

Wojciech Zaremba, Ilya Sutskever|arXiv (Cornell University)|Oct 17, 2014

Topic Modeling24 references319 citations

TL;DR

This paper demonstrates that Long Short-Term Memory (LSTM) networks can learn to evaluate simple, short programs requiring constant memory and linear time execution, using a novel curriculum learning strategy that significantly improves training performance. The key contribution is a curriculum method that outperforms both standard training and naive curriculum learning, enabling an LSTM to achieve 99% accuracy on adding two 9-digit numbers.

ABSTRACT

Recurrent Neural Networks (RNNs) with Long Short-Term Memory units (LSTM) are widely used because they are expressive and are easy to train. Our interest lies in empirically evaluating the expressiveness and the learnability of LSTMs in the sequence-to-sequence regime by training them to evaluate short computer programs, a domain that has traditionally been seen as too complex for neural networks. We consider a simple class of programs that can be evaluated with a single left-to-right pass using constant memory. Our main result is that LSTMs can learn to map the character-level representations of such programs to their correct outputs. Notably, it was necessary to use curriculum learning, and while conventional curriculum learning proved ineffective, we developed a new variant of curriculum learning that improved our networks' performance in all experimental conditions. The improved curriculum had a dramatic impact on an addition problem, making it possible to train an LSTM to add two 9-digit numbers with 99% accuracy.

Motivation & Objective

• To evaluate whether LSTMs can learn to execute short, simple programs involving arithmetic, variables, and control flow.
• To investigate the limitations of standard stochastic gradient descent (SGD) in training LSTMs for complex sequence-to-sequence tasks with long-term dependencies.
• To design and evaluate a new curriculum learning strategy that improves training efficiency and performance on difficult program evaluation tasks.
• To explore input transformations such as sequence reversal and replication to enhance LSTM performance on memorization and sequence learning tasks.

Proposed method

• The authors train LSTMs on a sequence-to-sequence framework where the input is a character-level string representation of a Python-like program and the output is the program's integer result.
• They define a restricted class of programs that can be evaluated in O(n) time and constant memory, including operations like addition, subtraction, multiplication, variable assignment, if-statements, and for-loops (without nested loops).
• A novel curriculum learning strategy is introduced that combines easy examples with a mix of harder examples to prevent memory pattern restructuring, avoiding performance degradation seen in naive curriculum learning.
• The method includes input transformations such as reversing the input sequence and replicating the input sequence to improve learning on memorization and sequence modeling tasks.
• The training process uses stochastic gradient descent (SGD) with backpropagation through time, and the model is evaluated on programs of varying lengths and nesting depths.
• The curriculum strategy is designed to gradually increase task difficulty while maintaining a balance between easy and hard examples to stabilize learning and avoid overfitting to simple patterns.

Experimental results

Research questions

• RQ1 Can LSTMs learn to evaluate simple programs with complex compositional structures, such as arithmetic operations and control flow, using only character-level input?
• RQ2 Why does naive curriculum learning fail in this task, and what structural limitations in the model's memory dynamics contribute to this failure?
• RQ3 Can a modified curriculum learning strategy that combines easy examples with a mixture of harder examples improve training stability and performance compared to standard training or naive curriculum?
• RQ4 How do input transformations like sequence reversal and replication affect the LSTM's ability to learn and memorize sequences in the program evaluation task?
• RQ5 To what extent does the model rely on memorization versus learning the correct algorithmic behavior for tasks like multi-digit addition?

Key findings

• The proposed curriculum learning strategy significantly outperforms both standard training and naive curriculum learning across all experimental conditions, enabling successful training on previously intractable tasks.
• The LSTM achieved 99% accuracy on the task of adding two 9-digit numbers, a task that was unattainable with standard SGD or naive curriculum learning.
• The combination of curriculum learning with input sequence reversal and replication further improved performance, particularly on memorization-intensive tasks.
• Naive curriculum learning was found to be harmful in this setting because it encouraged the model to overfit to simple patterns and restructure its memory representations when faced with harder examples.
• The model's performance was highly sensitive to training strategy, with the new curriculum method enabling convergence where previous approaches failed.
• The results suggest that the model's generalization capability is limited under identical training and test distributions, and that it may rely heavily on memorization rather than true algorithmic understanding, especially in the absence of distribution shift in evaluation data.