[Paper Review] TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Martı́n Abadi, Ashish Agarwal+38 more|arXiv (Cornell University)|Mar 14, 2016

Parallel Computing and Optimization Techniques42 references9,726 citations

TL;DR

TensorFlow is an interface and implementation for expressing machine learning algorithms that can run on mobile devices to large-scale distributed systems, using dataflow graphs and supporting training and inference; it is open-sourced under Apache 2.0.

ABSTRACT

TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms. A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards. The system is flexible and can be used to express a wide variety of algorithms, including training and inference algorithms for deep neural network models, and it has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields, including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction, and computational drug discovery. This paper describes the TensorFlow interface and an implementation of that interface that we have built at Google. The TensorFlow API and a reference implementation were released as an open-source package under the Apache 2.0 license in November, 2015 and are available at www.tensorflow.org.

Motivation & Objective

• Motivate the need for a unified system that spans mobile-to-cluster deployment of ML models.
• Describe the TensorFlow programming model, implementation, and system architecture.
• Show how TensorFlow enables scalable training and inference on diverse hardware.
• Demonstrate extensions and optimizations to improve performance and flexibility.
• Highlight practical deployments and usage in research and production contexts.

Proposed method

• Introduce a dataflow-style computation graph model with stateful nodes and control dependencies.
• Describe single-machine and distributed execution architectures, including device placement and cross-device communication via Send/Receive nodes.
• Explain automatic gradient computation and integration into the computation graph.
• Present extensions such as partial execution, device constraints, control flow, input operations, queues, and containers.
• Detail optimizations like common subexpression elimination, memory- and communication-conscious scheduling, and asynchronous kernels.
• Discuss fault tolerance, checkpointing, and recovery in distributed settings.

Experimental results

Research questions

• RQ1 How can a single framework express and execute a wide range of ML algorithms on heterogeneous hardware?
• RQ2 How can dataflow graph representations enable scalable, distributed training and inference?
• RQ3 How should device placement, cross-device communication, and memory management be handled to maximize performance and scalability?
• RQ4 What extensions (control flow, input handling, queues, containers) improve usability and efficiency for ML workloads?
• RQ5 What optimizations and fault-tolerance mechanisms are effective for large-scale ML systems?

Key findings

• TensorFlow provides a flexible programming model that maps dataflow graphs onto CPUs, GPUs, and other devices, enabling both research experimentation and production deployment.
• The system supports single-machine and distributed execution with scalable node placement, cross-device communication through Send/Receive nodes, and fault-tolerant checkpointing of variables.
• Automatic gradient computation is built into the graph, enabling common ML optimization methods like gradient descent.
• Various extensions (partial execution, control flow, input pipelines, queues, containers) enhance usability and performance for ML workloads.
• Optimizations include common subexpression elimination, ASAP/ALAP scheduling for reduce memory and communication overhead, and asynchronous kernel support.
• TensorFlow open-sourced under Apache 2.0 in November 2015 and has been used for research and production across many domains, including large-scale deep nets with hundreds of billions of parameters on hundreds of machines.