[Paper Review] The Serverless Scheduling Problem and NOAH

Manuel Stein|arXiv (Cornell University)|Sep 17, 2018

Scheduling and Optimization Algorithms6 references11 citations

TL;DR

This paper proposes NOAH, a noncooperative online allocation heuristic for serverless scheduling that balances response time and resource cost by tuning queue waiting time thresholds. By virtually allocating demand and controlling container churn, NOAH outperforms OpenWhisk and game-theoretic load balancing under high utilization, achieving lower response times and higher container utilization through configurable trade-offs between latency and cost.

ABSTRACT

The serverless scheduling problem poses a new challenge to Cloud service platform providers because it is rather a job scheduling problem than a traditional resource allocation or request load balancing problem. Traditionally, elastic cloud applications use managed virtual resource allocation and employ request load balancers to orchestrate the deployment. With serverless, the provider needs to solve both the load balancing and the allocation. This work reviews the current Apache OpenWhisk serverless event load balancing and a noncooperative game-theoretic load balancing approach for response time minimization in distributed systems. It is shown by simulation that neither performs well under high system utilization which inspired a noncooperative online allocation heuristic that allows tuning the trade-off between for response time and resource cost of each serverless function.

Motivation & Objective

• To address the challenge of balancing response time and resource cost in serverless platforms under high system utilization.
• To overcome limitations in existing schedulers like Apache OpenWhisk and noncooperative game-theoretic load balancing, which suffer from excessive container churn and poor performance at scale.
• To design a parameterizable online allocation heuristic that allows providers to tune scaling behavior based on application-specific response time and cost requirements.
• To evaluate the trade-off between container utilization, response time, and resource cost in a simulated distributed serverless environment.

Proposed method

• NOAH uses a noncooperative online allocation heuristic that virtually allocates demand to control instance scaling and reduce container churn.
• It applies the Erlang-C formula to estimate waiting times and configure the number of required hosts based on desired response time thresholds (α).
• The heuristic limits both concurrent events and total container count per host, preventing excessive resource allocation and idle container eviction.
• NOAH dynamically scales out only when necessary, covering the minimum required set of hosts and releasing unused VMs to reduce cost.
• It decouples load balancing from resource allocation, allowing independent tuning of response time and resource utilization.
• The approach is evaluated via simulation with varying arrival rates (Λ), waiting time thresholds (α), and container setup times (500 ms).

Experimental results

Research questions

• RQ1 How does NOAH compare to Apache OpenWhisk and noncooperative game-theoretic load balancing in terms of response time and container churn under high system utilization?
• RQ2 To what extent can NOAH reduce cold starts and improve container utilization by tuning the waiting time threshold α?
• RQ3 How does the trade-off between response time and resource cost manifest across different α settings in NOAH?
• RQ4 What is the impact of container setup time and message distribution on scheduling performance in serverless platforms?

Key findings

• NOAH maintains stable cold start rates proportional to workload increase, unlike OpenWhisk, which suffers from excessive churn and degraded performance at 50–60 messages per second.
• Under high load, NOAH with α = 10 μs (strict response time) achieves the lowest response times and highest container utilization, while α = 10 ms allows fewer hosts but higher churn.
• Container utilization is higher in NOAH than in OpenWhisk or noncooperative load balancing, especially under high load, due to reduced eviction and better instance reuse.
• The number of container creations in NOAH increases proportionally with workload, while OpenWhisk and noncoop show sharp spikes due to aggressive eviction and random load distribution.
• NOAH reduces resource cost by releasing unused hosts entirely, covering only the minimum required set of sites, unlike OpenWhisk and noncoop, which maintain full pool utilization.
• The simulation shows that allowing longer waiting times (α = 10 ms) reduces host coverage and increases churn, while stricter thresholds (α = 10 μs) improve utilization and reduce cold starts.