[Paper Review] An Uncertainty-Aware Approach To Optimal Configuration Of Stream Processing Systems

Pooyan Jamshidi, Giuliano Casale|arXiv (Cornell University)|Jun 22, 2016

Simulation Techniques and Applications25 references13 citations

TL;DR

This paper proposes BO4CO, a Bayesian optimization framework that uses Gaussian Processes to model uncertainty in stream processing system (SPS) performance and iteratively select optimal configurations under limited experimental budgets. It achieves at least a 10x improvement in performance over baseline methods by intelligently balancing exploration and exploitation in noisy, high-dimensional configuration spaces.

ABSTRACT

The datasets in this release support the results presented in the paper P. Jamshidi, G. Casale, "An Uncertainty-Aware Approach to Optimal Configuration of Stream Processing Systems", accepted for presentation at MASCOTS 2016. An open access to the paper is available at https://arxiv.org/abs/1606.06543 Also open source code is available at https://github.com/dice-project/DICE-Configuration-BO4CO The archive contains 10 comma separated datasets representing performance measurements (throughput and latency) for 3 different stream benchmark applications. These have been experimentally collected on 5 different cloud cluster over the course of 3 months (24/7). Each row in the datasets represents a different configuration setting for the application and the last two columns represent the average performance of the application measured over the course of 10 minutes under that specific configuration setting. The datasets contains a full factorial and exhaustive measurements for all possible settings limited to a predetermined interval for each variable. Each dataset is named in the following format: "<em>benchmark_application-dimensions-cluster_name</em>". For example, "wc-6d-c1" refers to WordCount benchmark application with 6 dimensions (i.e., we varied 6 configuration parameters) and the application was deployed on c1 cluster (OpenNebula, see Appendix). This resulted in a dataset of size 2880, i.e., it has taken 2880*10m=480h=20days for collecting the data! For more information about the data refer to the appendix of the paper: https://arxiv.org/abs/1606.06543. When referring to the dataset or code please cite the paper above.

Motivation & Objective

• To address the challenge of optimal configuration in stream processing systems (SPS) under limited experimental budgets and noisy measurements.
• To reduce reliance on heuristic, rule-based, or trial-and-error tuning methods that are slow and require expert knowledge.
• To develop an automated, uncertainty-aware configuration optimization framework that efficiently explores complex, non-linear configuration spaces.
• To enable accurate and fast convergence to near-optimal SPS configurations using sequential experimentation guided by probabilistic modeling.

Proposed method

• BO4CO employs Gaussian Processes (GPs) to model the response surface of SPS performance as a posterior distribution over configurations.
• It uses Bayesian optimization to sequentially select the next configuration to test based on acquisition functions that balance exploration and exploitation.
• The algorithm maintains a history of past experiments and updates the GP model with new data to refine uncertainty estimates and improve search efficiency.
• It supports both continuous and categorical configuration parameters, including integer, Boolean, and categorical variables like serialization methods or messaging frameworks.
• The method incorporates various kernel estimators to capture complex, non-linear interactions between configuration parameters.
• The acquisition function guides experimentation toward regions with high potential for optimal performance while minimizing uncertainty.

Experimental results

Research questions

• RQ1 Can a Bayesian optimization approach effectively reduce the number of experiments needed to find optimal configurations in stream processing systems?
• RQ2 How does uncertainty-aware configuration tuning improve performance compared to traditional heuristic or search-based methods?
• RQ3 To what extent can Gaussian Processes model the non-linear and noisy response surface of SPS performance under limited sampling?
• RQ4 Can the proposed method generalize across different SPS workloads and system configurations?

Key findings

• BO4CO achieved at least a 10x improvement in performance over baseline configuration algorithms in terms of distance to the optimal configuration.
• The method located optimal configurations within a limited experimental budget, demonstrating high sample efficiency on Apache Storm benchmarks.
• The learned GP model provided reliable performance predictions throughout the tuning process, enabling informed decision-making with minimal measurements.
• BO4CO outperformed five baseline approaches across three distinct SPS workloads, showing consistent superiority in convergence speed and accuracy.
• The algorithm guaranteed eventual discovery of optimal configurations by systematically reducing uncertainty in the response surface.