[Paper Review] Theano: new features and speed improvements

Frédéric Bastien, Pascal Lamblin+7 more|arXiv (Cornell University)|Nov 23, 2012

Parallel Computing and Optimization Techniques10 references1,038 citations

TL;DR

This paper presents significant new features and performance optimizations in Theano, a deep learning library that compiles symbolic mathematical expressions into efficient low-level code. By introducing the Scan operator for symbolic loops, enhancing gradient computation via the R-operator, and reducing Python overhead through optimizations like trust_input and batched function calls, Theano achieves competitive speed—matching or exceeding Torch7 and RNNLM on both CPU and GPU across various neural network benchmarks, especially with mini-batching.

ABSTRACT

Theano is a linear algebra compiler that optimizes a user's symbolically-specified mathematical computations to produce efficient low-level implementations. In this paper, we present new features and efficiency improvements to Theano, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.

Motivation & Objective

• To enhance Theano’s capabilities for training complex machine learning models, particularly recurrent neural networks, by introducing symbolic loop support.
• To reduce computational overhead in Theano, especially from Python interpreter and data type conversion, to improve execution speed.
• To enable efficient symbolic differentiation through loops and complex operations, supporting advanced optimization and training algorithms.
• To benchmark Theano’s performance against leading frameworks like Torch7 and RNNLM to establish its competitiveness in real-world deep learning scenarios.

Proposed method

• The Scan operator abstracts loops into a single node in the computational graph, enabling symbolic differentiation and optimization over variable-length iterations.
• The R-operator is implemented to compute Gauss-Newton Hessian-vector products efficiently, supporting second-order optimization methods.
• Theano’s compilation pipeline was optimized to reduce non-computational overhead, including disabling input type checks via the trust_input flag.
• Batched function calls (f.fn(n_calls=N)) were introduced to eliminate Python loop overhead when calling functions multiple times.
• The library leverages GPU acceleration via CUDA and integrates BLAS-optimized operations for dense tensor computations.
• Graph-level optimizations such as loop fusion and numerical stability improvements (e.g., stable log-sigmoid) are applied automatically during compilation.

Experimental results

Research questions

• RQ1 Can Theano efficiently support recurrent models through a symbolic loop abstraction without sacrificing differentiability or performance?
• RQ2 How does Theano’s performance compare to Torch7 and RNNLM on standard deep learning benchmarks, especially with varying batch sizes?
• RQ3 To what extent do input type checking and Python interpreter overhead limit Theano’s speed, and can these be mitigated?
• RQ4 Can Theano’s symbolic differentiation capabilities be extended to support complex operations like the R-operator and backpropagation through loops?

Key findings

• Theano outperforms Torch7 on CPU for models with hidden layers when using mini-batches, due to efficient BLAS-based parallelism.
• With mini-batch sizes of 10 or 60, Theano matches or exceeds Torch7’s speed across all tested architectures, especially as computation becomes dominant over overhead.
• On GPU with batch size 1, RNNLM is faster than Theano for smaller models, but Theano closes the gap significantly on larger models due to reduced overhead in optimized configurations.
• Disabling input type checks via trust_input improves performance by reducing data conversion overhead, especially when inputs are already correct.
• Batched function calls (f.fn(n_calls=N)) eliminate Python loop overhead and significantly speed up iterative training processes where only parameter updates matter.
• The Scan operator enables correct and efficient gradient computation through loops, including backpropagation through time and R-operator evaluation, even for variable-length iterations.