[Paper Review] A Review of Deep Reinforcement Learning in Serverless Computing: Function Scheduling and Resource Auto-Scaling

Amjad Yousef Majid, Eduard Marin|arXiv (Cornell University)|Oct 5, 2023

IoT and Edge/Fog Computing4 citations

TL;DR

This paper reviews deep reinforcement learning (DRL) applications in serverless computing, focusing on function scheduling and resource auto-scaling. It evaluates DRL’s ability to dynamically optimize performance and cost in real-time, identifying promising results but highlighting challenges like cold starts, simulation realism, and multi-objective trade-offs. The study calls for robust DRL models, standardized benchmarks, and exploration of multi-agent and federated learning for scalable, privacy-preserving solutions.

ABSTRACT

In the rapidly evolving field of serverless computing, efficient function scheduling and resource scaling are critical for optimizing performance and cost. This paper presents a comprehensive review of the application of Deep Reinforcement Learning (DRL) techniques in these areas. We begin by providing an overview of serverless computing, highlighting its benefits and challenges, with a particular focus on function scheduling and resource scaling. We then delve into the principles of deep reinforcement learning (DRL) and its potential for addressing these challenges. A systematic review of recent studies applying DRL to serverless computing is presented, covering various algorithms, models, and performances. Our analysis reveals that DRL, with its ability to learn and adapt from an environment, shows promising results in improving the efficiency of function scheduling and resource scaling in serverless computing. However, several challenges remain, including the need for more realistic simulation environments, handling of cold starts, and the trade-off between learning time and scheduling performance. We conclude by discussing potential future directions for this research area, emphasizing the need for more robust DRL models, better benchmarking methods, and the exploration of multi-agent reinforcement learning for more complex serverless architectures. This review serves as a valuable resource for researchers and practitioners aiming to understand and advance the application of DRL in serverless computing.

Motivation & Objective

• To systematically review recent studies applying deep reinforcement learning (DRL) to function scheduling and resource auto-scaling in serverless computing.
• To identify key DRL algorithms, models, and performance metrics used in optimizing serverless workloads.
• To analyze the challenges in applying DRL to real-world serverless environments, including cold starts, simulation fidelity, and hyperparameter sensitivity.
• To propose future research directions for advancing DRL in serverless systems, focusing on multi-objective optimization, robustness, and privacy-preserving learning.
• To establish a foundation for benchmarking and standardizing DRL evaluation in serverless computing through improved metrics and simulation environments.

Proposed method

• Conduct a systematic literature review of recent DRL applications in serverless computing, focusing on function scheduling and resource scaling.
• Categorize and analyze DRL algorithms used in the reviewed studies, including DQN, A3C, PPO, and DDPG, based on architecture and training methodology.
• Evaluate performance using standard metrics such as cost, latency, throughput, and resource utilization, comparing results across studies.
• Assess the realism and scalability of simulation environments used in experiments, identifying gaps in modeling real-world dynamics like cold starts.
• Integrate insights from multi-agent reinforcement learning and federated learning to propose collaborative, privacy-preserving DRL frameworks.
• Propose methodological improvements, including standardized benchmarks, uncertainty quantification, and interpretability tools for DRL policies in production settings.

Figure 1. Deep Reinforcement Learning Overview

Experimental results

Research questions

• RQ1 Which DRL algorithms are most effective for function scheduling and resource auto-scaling in serverless environments, and how do they compare in terms of performance and adaptability?
• RQ2 What are the key challenges in deploying DRL models in real-world serverless systems, particularly regarding cold starts, simulation fidelity, and training time?
• RQ3 How can DRL models be made more robust to dynamic and uncertain workloads in production serverless deployments?
• RQ4 What role can multi-objective and federated DRL play in balancing cost, performance, and privacy in distributed serverless architectures?
• RQ5 What standardized benchmarks and evaluation metrics are needed to fairly compare DRL-based solutions in serverless computing?

Key findings

• DRL demonstrates strong potential for optimizing function scheduling and resource auto-scaling in serverless computing, achieving improved cost and performance trade-offs compared to traditional methods.
• Existing DRL models often focus on single or dual objectives (e.g., cost and latency), leaving multi-objective optimization—balancing cost, performance, and resource utilization—as an open challenge.
• Simulation environments used in current studies lack realism, particularly in modeling cold starts and dynamic workload variations, limiting the generalizability of results.
• DRL models exhibit high sensitivity to hyperparameter settings and require significant computational resources, posing practical barriers to deployment in production systems.
• There is a lack of standardized benchmarks and evaluation metrics, making cross-study comparisons difficult and hindering progress in model selection and validation.
• Future directions such as federated learning and uncertainty-aware DRL (e.g., Bayesian networks, ensemble methods) are critical for enabling privacy-preserving, robust, and adaptive serverless orchestration.