[Paper Review] On-demand Cold Start Frequency Reduction with Off-Policy Reinforcement Learning in Serverless Computing

Siddharth Agarwal, Maria A. Rodriguez+1 more|arXiv (Cornell University)|Aug 15, 2023

Cloud Computing and Resource Management4 citations

TL;DR

This paper proposes a Q-learning-based reinforcement learning agent to proactively reduce cold starts in serverless computing by dynamically scaling function instances using CPU utilization, available instances, and response failure rate. Evaluated on Kubeless with a matrix multiplication workload, the agent improves throughput by up to 8.81% and reduces resource wastage by up to 37% compared to default and keep-alive policies.

ABSTRACT

Function-as-a-Service (FaaS) is a cloud computing paradigm offering an event-driven execution model to applications. It features serverless attributes by eliminating resource management responsibilities from developers, and offers transparent and on-demand scalability of applications. To provide seamless on-demand scalability, new function instances are prepared to serve the incoming workload in the absence or unavailability of function instances. However, FaaS platforms are known to suffer from cold starts, where this function provisioning process introduces a non-negligible delay in function response and reduces the end-user experience. Therefore, the presented work focuses on reducing the frequent, on-demand cold starts on the platform by using Reinforcement Learning(RL). The proposed approach uses model-free Q-learning that consider function metrics such as CPU utilization, existing function instances, and response failure rate, to proactively initialize functions, in advance, based on the expected demand. The proposed solution is implemented on Kubeless and evaluated using an open-source function invocation trace applied to a matrix multiplication function. The evaluation results demonstrate a favourable performance of the RL-based agent when compared to Kubeless' default policy and a function keep-alive policy by improving throughput by up to 8.81% and reducing computation load and resource wastage by up to 55% and 37%, respectively, that is a direct outcome of reduced cold starts.

Motivation & Objective

• Address the performance degradation caused by frequent cold starts in Function-as-a-Service (FaaS) platforms, which delay response times and reduce throughput.
• Overcome the limitations of reactive, threshold-based auto-scaling (e.g., Kubernetes HPA) that trigger cold starts only after demand spikes.
• Develop a model-free, off-policy RL agent that learns to anticipate workload demand and pre-initialize function instances to minimize cold starts.
• Improve platform efficiency by reducing resource wastage and increasing request success rates through proactive scaling.

Proposed method

• Design a Q-learning agent that observes the environment state using metrics: average CPU utilization, number of existing function instances, and response failure rate.
• Define a custom reward function that penalizes high failure rates and excessive resource usage while rewarding stable performance and low cold start frequency.
• Model the RL environment on the Kubeless serverless framework, simulating real-world function invocation patterns using Apache JMeter.
• Use discrete state discretization and lazy loading of dependencies to manage state space complexity and avoid state explosion.
• Train the agent over 500 epochs using trial-and-error interaction with the environment, learning optimal scaling actions without prior knowledge of workload patterns.
• Evaluate the agent against baseline policies: default Kubernetes HPA and a function keep-alive policy, using throughput, resource usage, and failure rate as key metrics.

Figure 1 : System Model

Experimental results

Research questions

• RQ1 Can a Q-learning agent effectively reduce cold start frequency in serverless environments by proactively scaling function instances based on real-time metrics?
• RQ2 How does the performance of the RL-based agent compare to reactive HPA and function keep-alive policies in terms of throughput and resource efficiency?
• RQ3 To what extent do CPU utilization, available instances, and failure rate influence the agent’s ability to predict and prevent cold starts?
• RQ4 Does the RL agent generalize across varying workloads and request patterns without retraining, or is it sensitive to specific demand characteristics?

Key findings

• The RL-based agent improved platform throughput by up to 8.81% compared to the default HPA policy and the function keep-alive policy.
• Resource wastage was reduced by up to 37% due to fewer unnecessary function instances being created, directly resulting from reduced cold starts.
• The agent achieved a significant reduction in request failure rate by proactively scaling function instances before demand spikes, improving service reliability.
• The reward function effectively guided the agent to balance between over-provisioning and under-provisioning, minimizing both resource waste and cold start occurrences.
• The agent outperformed both baselines across all evaluation metrics, demonstrating the effectiveness of off-policy, model-free RL in proactive cold start mitigation.
• Despite limited training iterations (500 epochs), the agent learned to compensate for failures in subsequent steps by leveraging acquired knowledge, indicating robustness in learning.

Figure 2 : Function Warm Start & Cold Start workflow