[Paper Review] A Data-Driven Approach to Dynamically Adjust Resource Allocation for Compute Clusters

Francesco Pace, Dimitrios Milios+3 more|ArXiv.org|Jul 1, 2018

Cloud Computing and Resource Management30 references4 citations

TL;DR

This paper proposes a data-driven, dynamic resource allocation mechanism for compute clusters that forecasts application resource utilization using Gaussian process regression and adjusts allocations in real time to reduce resource slack. By incorporating prediction uncertainty and a pessimistic preemption policy, the system cuts average turnaround time by over 50% and reduces resource slack by ~40% without application failures, significantly improving cluster efficiency and responsiveness.

ABSTRACT

Nowadays, data-centers are largely under-utilized because resource allocation is based on reservation mechanisms which ignore actual resource utilization. Indeed, it is common to reserve resources for peak demand, which may occur only for a small portion of the application life time. As a consequence, cluster resources often go under-utilized. In this work, we propose a mechanism that improves cluster utilization, thus decreasing the average turnaround time, while preventing application failures due to contention in accessing finite resources such as RAM. Our approach monitors resource utilization and employs a data-driven approach to resource demand forecasting, featuring quantification of uncertainty in the predictions. Using demand forecast and its confidence, our mechanism modulates cluster resources assigned to running applications, and reduces the turnaround time by more than one order of magnitude while keeping application failures under control. Thus, tenants enjoy a responsive system and providers benefit from an efficient cluster utilization.

Motivation & Objective

• To address the widespread underutilization of data-center resources caused by static, reservation-based allocation in cloud clusters.
• To reduce resource slack—defined as the gap between allocated and actual utilization—without increasing application failure risk.
• To improve system responsiveness and cluster utilization by dynamically adjusting resource allocations based on real-time demand forecasts.
• To design a mechanism that balances prediction uncertainty with system safety, avoiding failures due to insufficient resources like RAM.
• To validate the approach in a real testbed using representative workloads from Apache Spark and TensorFlow applications.

Proposed method

• Uses Gaussian Process (GP) regression to model and forecast application resource utilization (CPU, RAM) over time, capturing temporal dynamics and uncertainty.
• Incorporates prediction confidence intervals to guide allocation adjustments, ensuring safety margins against under-provisioning.
• Employs a pessimistic preemption policy that reallocates resources only when safe, minimizing failure risk during dynamic adjustments.
• Monitors real-time resource usage across cluster nodes and correlates it with application behavior to train and update the forecasting model.
• Integrates with a FIFO scheduler to compare static reservation-based allocation against dynamic shaping in controlled experiments.
• Uses a testbed with 10 servers and real workloads (Spark and TensorFlow) to evaluate performance under realistic conditions.

Experimental results

Research questions

• RQ1 How can resource utilization forecasts be made accurate and uncertainty-aware to enable safe dynamic allocation in shared clusters?
• RQ2 What is the impact of dynamic resource adjustment on cluster turnaround time and resource slack compared to static reservation-based allocation?
• RQ3 How can prediction errors be mitigated to prevent application failures, especially for non-shareable resources like RAM?
• RQ4 What preemption strategy best balances responsiveness and reliability in a dynamic allocation system?
• RQ5 To what extent can dynamic allocation improve cluster utilization without increasing failure rates?

Key findings

• The dynamic allocation mechanism reduced resource slack by approximately 40% compared to the baseline static reservation system.
• Median application turnaround time was reduced by about 50% due to shorter queueing times and more responsive resource provisioning.
• The system achieved zero application or component failures across all experiments, even under high load, when using the pessimistic preemption policy.
• The use of uncertainty-aware forecasting enabled safe, aggressive resource adjustments that minimized idle capacity without compromising reliability.
• The approach demonstrated substantial improvements in system responsiveness and efficiency, especially in handling mixed workloads of elastic and rigid applications.
• The results were validated in a real testbed with real workloads, confirming the scalability and robustness of the proposed mechanism.