[Paper Review] C3O: Collaborative Cluster Configuration Optimization for Distributed Data Processing in Public Clouds

Jonathan Will, Lauritz Thamsen+3 more|arXiv (Cornell University)|Jul 28, 2021

Cloud Computing and Resource Management24 references17 citations

TL;DR

C3O is a collaborative runtime prediction system that optimizes cluster configurations for distributed data processing in public clouds by leveraging shared historical execution data from diverse users. It uses context-aware regression models trained on globally distributed job metrics, achieving a mean absolute error below 3% in predicting job runtimes across 930 Spark jobs, significantly outperforming single-user approaches.

ABSTRACT

Distributed dataflow systems enable data-parallel processing of large datasets on clusters. Public cloud providers offer a large variety and quantity of resources that can be used for such clusters. Yet, selecting appropriate cloud resources for dataflow jobs - that neither lead to bottlenecks nor to low resource utilization - is often challenging, even for expert users such as data engineers. We present C3O, a collaborative system for optimizing data processing cluster configurations in public clouds based on shared historical runtime data. The shared data is utilized for predicting the runtimes of data processing jobs on different possible cluster configurations, using specialized regression models. These models take the diverse execution contexts of different users into account and exhibit mean absolute errors below 3% in our experimental evaluation with 930 unique Spark jobs.

Motivation & Objective

• To address the challenge of selecting optimal cloud resources for distributed data processing jobs without incurring high provisioning or profiling overhead.
• To enable collaborative performance modeling across users by sharing historical runtime data from diverse execution contexts.
• To develop a runtime predictor that improves prediction accuracy by leveraging globally collected, context-aware training data.
• To reduce resource over-provisioning and improve cost-efficiency in public cloud environments for sporadic data processing workloads.

Proposed method

• C3O collects and shares historical runtime metrics (e.g., job duration, resource usage) from users across different execution contexts, including varying data sizes, cluster configurations, and workloads.
• It employs a black-box runtime prediction model that dynamically selects from multiple regression models based on expected accuracy in a given scenario.
• The system uses cross-validation to select the most promising model from available training data, ensuring robustness across diverse configurations.
• The predictor is trained on shared data from 930 unique Spark jobs, incorporating contextual features such as data size, number of partitions, and hardware characteristics.
• It supports both collaborative and single-user modes, allowing users to benefit from global data when available and fall back to local data when necessary.
• The architecture enables dynamic model switching at runtime, adapting to new configurations and minimizing prediction error.

Experimental results

Research questions

• RQ1 Can shared historical runtime data from diverse users improve the accuracy of runtime prediction for distributed data processing jobs in public clouds?
• RQ2 How does the performance of a collaborative runtime predictor compare to single-user models trained only on local data?
• RQ3 To what extent can context-aware regression models reduce prediction error in cluster configuration optimization?
• RQ4 What is the impact of cross-validation and model selection strategies on prediction accuracy in heterogeneous execution environments?

Key findings

• The C3O predictor achieves a mean absolute error of less than 3% in predicting job runtimes across 930 unique Spark jobs, demonstrating high accuracy.
• Collaborative training with globally shared data significantly improves prediction accuracy compared to models trained on local-only data.
• The system maintains strong performance even with limited local data, showing that global data enhances prediction reliability from the outset.
• Cross-validation-based model selection effectively identifies the most accurate regression model for a given configuration and context.
• The approach reduces the need for costly profiling runs and over-provisioning, especially beneficial in cloud environments with long cluster provisioning times.
• The system is effective for both collaborative and non-collaborative use cases, supporting organizations with or without access to shared data.