[Paper Review] DyNet: The Dynamic Neural Network Toolkit

Graham Neubig, Chris Dyer+23 more|arXiv (Cornell University)|Jan 15, 2017

Topic Modeling48 references344 citations

TL;DR

DyNet introduces a dynamic declaration neural network toolkit where computation graphs are constructed on the fly during execution, enabling variable architectures per input and fast prototyping.

ABSTRACT

We describe DyNet, a toolkit for implementing neural network models based on dynamic declaration of network structure. In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives. In DyNet's dynamic declaration strategy, computation graph construction is mostly transparent, being implicitly constructed by executing procedural code that computes the network outputs, and the user is free to use different network structures for each input. Dynamic declaration thus facilitates the implementation of more complicated network architectures, and DyNet is specifically designed to allow users to implement their models in a way that is idiomatic in their preferred programming language (C++ or Python). One challenge with dynamic declaration is that because the symbolic computation graph is defined anew for every training example, its construction must have low overhead. To achieve this, DyNet has an optimized C++ backend and lightweight graph representation. Experiments show that DyNet's speeds are faster than or comparable with static declaration toolkits, and significantly faster than Chainer, another dynamic declaration toolkit. DyNet is released open-source under the Apache 2.0 license and available at http://github.com/clab/dynet.

Motivation & Objective

• Motivate the need for rapid prototyping and flexible architectures in deep learning tooling.
• Propose a dynamic declaration programming model that unifies declaration and execution of computation graphs.
• Show that dynamic graph construction can be efficiently implemented with a lightweight backend.
• Demonstrate use cases such as recurrent and tree-structured networks in NLP tasks.

Proposed method

• Introduce dynamic declaration as an alternative to static computation graphs used in Theano, TensorFlow, and CNTK.
• Implement a lightweight C++ backend with an implicit computation graph constructed during forward evaluation.
• Provide core components: Parameter, LookupParameter, Model, Expression, and Trainer, with backward propagation for learning.
• Support dynamic network structures, recursion, and flow control via host language (C++/Python) while keeping graph types simple.
• Offer native support for RNNs, tree-structured networks, mini-batching, and data-parallel multi-processing.
• Present high-level programming examples and two dynamic graph construction demos (TreeRNN and dynamic flow control).

Experimental results

Research questions

• RQ1 Can dynamic graph construction per training example match or exceed the efficiency of static declaration toolkits?
• RQ2 How does dynamic declaration facilitate implementing variable-sized or structure inputs (e.g., sequences, trees, graphs) and dynamic flow control?
• RQ3 What is the impact of a unified dynamic programming model on rapid prototyping and debugging of neural models?

Key findings

• DyNet achieves execution speeds comparable to static-declaration toolkits and faster than Chainer in experiments.
• Dynamic declaration simplifies implementing dynamic architectures such as tree-structured networks and variable-length inputs.
• The toolkit provides native support for recurrent and tree-structured networks and efficient mini-batching and multi-processing.
• Graph construction overhead is minimized via an optimized C++ backend and lightweight graph representation.
• DyNet is open-source under the Apache 2.0 license.