[Paper Review] Scalable, Fast Cloud Computing with Execution Templates

Omid Mashayekhi, Hang Qu+2 more|arXiv (Cornell University)|Jun 6, 2016

Cloud Computing and Resource Management21 references3 citations

TL;DR

This paper introduces execution templates, a control plane abstraction that caches scheduling decisions for repetitive, CPU-bound workloads in cloud computing. By treating repeated computation loops as reusable templates, the system reduces scheduling overhead and enables 16-43x speedups over Spark and Naiad, scaling to 100 nodes (800 cores), even for tasks as short as 100μs.

ABSTRACT

Large scale cloud data analytics applications are often CPU bound. Most of these cycles are wasted: benchmarks written in C++ run 10-51 times faster than frameworks such as Naiad and Spark. However, calling faster implementations from those frameworks only sees moderate (3-5x) speedups because their control planes cannot schedule work fast enough. This paper presents execution templates, a control plane abstraction for CPU-bound cloud applications, such as machine learning. Execution templates leverage highly repetitive control flow to cache scheduling decisions as {\it templates}. Rather than reschedule hundreds of thousands of tasks on every loop execution, nodes instantiate these templates. A controller's template specifies the execution across all worker nodes, which it partitions into per-worker templates. To ensure that templates execute correctly, controllers dynamically patch templates to match program control flow. We have implemented execution templates in Nimbus, a C++ cloud computing framework. Running in Nimbus, analytics benchmarks can run 16-43 times faster than in Naiad and Spark. Nimbus's control plane can scale out to run these faster benchmarks on up to 100 nodes (800 cores).

Motivation & Objective

• Address the performance bottleneck in modern cloud analytics frameworks caused by slow control planes when running highly optimized, CPU-bound code.
• Overcome the limitation that existing frameworks like Spark and Naiad cannot scale beyond 20 nodes when running C++-optimized kernels due to controller saturation.
• Enable large-scale, high-throughput execution of fine-grained tasks (down to 100μs) in cloud environments by abstracting repetitive control flow into reusable scheduling templates.
• Design a new control plane abstraction that allows centralized controllers to manage extremely high task rates—up to 100x faster than prior systems—without becoming a bottleneck.
• Support complex, real-world workloads such as high-performance simulations (e.g., PhysBAM) with minimal runtime overhead and near-native performance.

Proposed method

• Introduce execution templates as a control plane abstraction that cache scheduling decisions for repeated loops in CPU-bound applications.
• Decompose a job’s control flow into a controller-level template and per-worker templates, partitioning work across nodes while preserving dependencies.
• Use dynamic program analysis to patch templates at runtime to handle variations in control and data flow, ensuring correctness despite dynamic behavior.
• Implement the abstraction in Nimbus, a C++-based analytics framework that uses mutable data objects to support in-place computation and reduce memory overhead.
• Leverage a single controller to issue a single message to instantiate thousands of tasks via a template, drastically reducing scheduling frequency and latency.
• Optimize task scheduling for sub-millisecond tasks (down to 100μs), enabling workloads previously limited to HPC or MPI to run efficiently in the cloud.

Experimental results

Research questions

• RQ1 Why do modern cloud frameworks like Spark and Naiad achieve only modest speedups (3–5x) when running C++-optimized kernels, despite the kernels being up to 51x faster?
• RQ2 What causes the control plane to become a bottleneck when scaling CPU-optimized workloads across many nodes in existing frameworks?
• RQ3 Can a control plane abstraction be designed to handle extremely high task rates (e.g., 1000 tasks per millisecond) without sacrificing correctness or scalability?
• RQ4 How can scheduling decisions be cached and reused across repeated iterations of a loop in a way that remains correct under dynamic control flow?
• RQ5 To what extent can a cloud framework with such an abstraction support real-world, complex workloads like high-performance simulations with microsecond-level tasks?

Key findings

• C++ implementations of machine learning benchmarks run up to 51 times faster than their Spark counterparts, but calling these kernels via JNI only yields 3–5x speedups due to control plane bottlenecks.
• Spark and Naiad’s control planes become saturated when scaling beyond 20 worker nodes, causing completion times to increase rather than decrease, due to scheduling overhead.
• With execution templates, Nimbus achieves 16–43x speedup over Spark and 16–23x over Naiad on the same optimized benchmarks, while scaling to 100 nodes (800 cores).
• Execution templates enable the controller to handle tasks as short as 100μs—100 times shorter than what prior systems support—by caching scheduling decisions and reducing scheduling frequency.
• The PhysBAM simulation library, originally hand-tuned for MPI, runs within 15% of native MPI performance in Nimbus when using execution templates, demonstrating real-world applicability.
• Dynamic patching of templates at runtime ensures correctness even when control or data flow varies, allowing the abstraction to work across diverse, complex workloads without sacrificing performance.