[Paper Review] TensorFlow-Serving: Flexible, High-Performance ML Serving

Christopher Olston, Noah Fiedel+7 more|arXiv (Cornell University)|Dec 17, 2017

Machine Learning and Data ClassificationComputer Science7 references95 citations

TL;DR

TensorFlow-Serving provides a flexible, high-performance ML serving framework with a library, a canonical binary, and a hosted service (TFS 2) for efficient model lifecycle management, batching, and multi-model hosting in Google and Google Cloud.

ABSTRACT

We describe TensorFlow-Serving, a system to serve machine learning models inside Google which is also available in the cloud and via open-source. It is extremely flexible in terms of the types of ML platforms it supports, and ways to integrate with systems that convey new models and updated versions from training to serving. At the same time, the core code paths around model lookup and inference have been carefully optimized to avoid performance pitfalls observed in naive implementations. Google uses it in many production deployments, including a multi-tenant model hosting service called TFS^2.

Motivation & Objective

• Motivate the need for production-ready ML model serving infrastructure.
• Describe the architectural design that supports multiple ML platforms and model lifecycles.
• Present mechanisms for safe model upgrades, canary and rollback, and efficient memory management.
• Explain the hosted service (TFS 2) and how it automates model deployment and routing.

Proposed method

• Describe a three-layer design: C++ library, canonical server binary, and hosted service.
• Implement model lifecycle management with Sources, Source Routers, Source Adapters, and a Manager using an aspired versions API.
• Introduce AspiredVersionsManager with availability- or resource-preserving transition policies and tail-latency optimizations.
• Provide multiple inference APIs, including low-level tensor interfaces and tf.Example-based higher-level APIs, with logging for debugging and quality checks.
• Develop inter-request batching with a core batching library that supports multiple queues and dynamic serving of models/versions.
• Offer a canonical binary deployment and a hosted service (TFS 2) for easier usage and automatic best-practices enforcement.

Experimental results

Research questions

• RQ1 How can we design a general ML model serving system that is agnostic to the underlying ML framework?
• RQ2 How should model versions be loaded, switched, and rolled back with minimal latency and safe canary testing?
• RQ3 What batching and threading strategies enable high throughput while keeping tail latency low for GPU/TPU inference?
• RQ4 How can serving infrastructure be offered as a hosted service with automated routing and resource management?
• RQ5 What mechanisms ensure end-to-end ML pipeline quality checks before serving new versions?

Key findings

• The library, binary, and hosted service cover diverse deployment needs, including multi-model and multi-tenant setups.
• AspiredVersionsManager enables canary and rollback workflows to safely validate new versions before full rollout.
• TensorFlow-Serving can handle about 100,000 requests per second per core when RPC/TensorFlow layers are excluded from the measurement.
• Google-scale adoption includes hundreds of projects and tens of millions of inferences per second across users.
• TFS 2 automates model assignment to serving jobs, canary/rollback, and relies on Spanner for global state with hedged requests to mitigate latency spikes.