[Paper Review] Task Bench: A Parameterized Benchmark for Evaluating Parallel Runtime Performance

Elliott Slaughter, Wei Wu+13 more|arXiv (Cornell University)|Aug 15, 2019

Parallel Computing and Optimization Techniques35 references4 citations

TL;DR

Task Bench is a parameterized benchmark framework that decouples benchmark specification from system implementations, enabling efficient comparison of 15 parallel and distributed programming systems across diverse workloads. It introduces the Minimum Effective Task Granularity (METG) metric, revealing that 100µs is a practical lower bound for efficient task execution at scale with current technologies, and exposes over five orders of magnitude variation in system overheads.

ABSTRACT

We present Task Bench, a parameterized benchmark designed to explore the performance of parallel and distributed programming systems under a variety of application scenarios. Task Bench lowers the barrier to benchmarking multiple programming systems by making the implementation for a given system orthogonal to the benchmarks themselves: every benchmark constructed with Task Bench runs on every Task Bench implementation. Furthermore, Task Bench's parameterization enables a wide variety of benchmark scenarios that distill the key characteristics of larger applications. We conduct a comprehensive study with implementations of Task Bench in 15 programming systems on up to 256 Haswell nodes of the Cori supercomputer. We introduce a novel metric, minimum effective task granularity to study the baseline runtime overhead of each system. We show that when running at scale, 100 μs is the smallest granularity that even the most efficient systems can reliably support with current technologies. We also study each system's scalability, ability to hide communication and mitigate load imbalance.

Motivation & Objective

• To address the lack of comprehensive, fair, and scalable performance comparisons among parallel and distributed programming systems.
• To reduce the engineering overhead of benchmarking multiple systems by decoupling benchmark logic from system-specific implementations.
• To quantify the impact of runtime system overheads on application performance, especially at fine-grained task levels.
• To introduce a new metric, Minimum Effective Task Granularity (METG), that isolates and measures the true cost of task runtime systems.
• To enable systematic study of scalability, communication hiding, and load imbalance mitigation across diverse systems.

Proposed method

• Task Bench models applications as directed task graphs with configurable computation, communication, and dependencies, enabling diverse workload patterns.
• It abstracts system-specific implementation via a core API, allowing any system implementing the interface to run all benchmarks with minimal porting effort.
• The framework supports heterogeneous task graphs, concurrent execution, and variable payload sizes to model real-world application characteristics.
• A novel metric, METG(50%), is defined as the smallest task granularity achieving at least 50% of peak performance, enabling fair comparison of system overheads.
• The benchmark is evaluated on 256-node Cori supercomputer across 15 diverse systems, including MPI, Spark, Dask, and task-based runtimes.
• System implementations are built using a standardized interface, ensuring orthogonal, composable, and reproducible evaluations.

Experimental results

Research questions

• RQ1 What is the minimum task granularity at which a parallel programming system can achieve at least 50% of peak performance, and how does this vary across systems?
• RQ2 How do different programming models and runtimes perform under varying workloads, including communication-heavy, compute-intensive, and load-imbalanced scenarios?
• RQ3 To what extent do system-level overheads dominate application performance, and how can they be isolated and measured?
• RQ4 How do features like asynchronous execution and static vs. dynamic task scheduling affect scalability and efficiency?
• RQ5 What is the impact of system choice on performance when scaling to large node counts (e.g., 256 nodes)?

Key findings

• The Minimum Effective Task Granularity (METG) for current distributed programming systems varies by over five orders of magnitude, highlighting extreme differences in runtime overhead.
• At scale (256 nodes), 100µs is a practical lower bound for efficient task execution, with no system achieving efficient performance below this threshold.
• Systems supporting asynchronous execution show performance benefits under load imbalance and balanced workloads, but high baseline overheads can negate these gains.
• Task-based systems relying on dynamic parallelism discovery suffer from sequential bottlenecks that limit scaling, even with dynamic pruning techniques.
• Static, compile-time task analysis can fully mitigate the sequential bottleneck, enabling better scaling than dynamic approaches.
• Large-scale data analytics systems like Spark require tasks of tens of seconds to scale beyond small node counts, indicating coarse-grained tasking is standard in such workloads.