[Paper Review] KeystoneML: Optimizing Pipelines for Large-Scale Advanced Analytics

Evan Sparks, Shivaram Venkataraman+3 more|arXiv (Cornell University)|Oct 29, 2016

Advanced Neural Network Applications42 references4 citations

TL;DR

KeystoneML is a high-level, type-safe framework for end-to-end optimization of large-scale machine learning pipelines in distributed environments. By combining cost-based physical operator selection with whole-pipeline optimizations—including intelligent intermediate data materialization—it achieves up to 15× higher training throughput than unoptimized execution on real-world image classification, speech, and NLP workloads.

ABSTRACT

Modern advanced analytics applications make use of machine learning techniques and contain multiple steps of domain-specific and general-purpose processing with high resource requirements. We present KeystoneML, a system that captures and optimizes the end-to-end large-scale machine learning applications for high-throughput training in a distributed environment with a high-level API. This approach offers increased ease of use and higher performance over existing systems for large scale learning. We demonstrate the effectiveness of KeystoneML in achieving high quality statistical accuracy and scalable training using real world datasets in several domains. By optimizing execution KeystoneML achieves up to 15x training throughput over unoptimized execution on a real image classification application.

Motivation & Objective

• Address the lack of end-to-end optimization in existing ML systems for large-scale, multi-stage analytics pipelines.
• Enable seamless scaling of ML pipelines across changing data volumes and hardware configurations.
• Automate the selection of optimal physical operators and execution strategies based on data characteristics and resource constraints.
• Improve performance and statistical accuracy by optimizing both individual operators and the full pipeline workflow.

Proposed method

• Expose a high-level, type-safe API using logical operators to express complete ML workflows end-to-end.
• Design a cost-based optimizer that accounts for both computation and communication costs in distributed execution.
• Formulate an intermediate data materialization problem to reduce redundant computation in iterative ML pipelines.
• Develop a greedy algorithm to efficiently identify which intermediate states to cache for performance gains.
• Integrate domain-agnostic optimizations with awareness of data properties like sparsity and dimensionality.
• Support dynamic reconfiguration of pipelines to adapt to changes in data size, hardware, and feature sets.

Experimental results

Research questions

• RQ1 How can end-to-end ML pipelines be automatically optimized for performance and scalability in distributed environments?
• RQ2 What role do data characteristics—such as sparsity and dimensionality—play in selecting optimal physical execution plans for ML operators?
• RQ3 To what extent can cost-based optimization of physical operators and whole-pipeline strategies improve training throughput compared to naive execution?
• RQ4 Can a unified system achieve both high-level abstraction and low-level performance through integrated optimization?
• RQ5 How does KeystoneML’s optimization strategy compare to specialized systems in terms of resource efficiency and statistical accuracy?

Key findings

• KeystoneML achieves up to 15× higher training throughput compared to unoptimized execution, primarily through combined physical and whole-pipeline optimizations.
• End-to-end optimization alone improves performance by up to 7×, demonstrating the value of holistic pipeline tuning.
• Poor physical operator selection can lead to a 260× performance slowdown, highlighting the critical role of intelligent operator choice.
• On a real image classification pipeline with over 1 million images, KeystoneML shows linear scalability across hundreds of machines.
• KeystoneML matches the performance of a specialized phoneme classification system on a BlueGene supercomputer while using 8× fewer resources.
• The system achieves statistical performance comparable to state-of-the-art results in image classification, sentiment analysis, and speech recognition.