[Paper Review] Dynamic SDN-based Radio Access Network Slicing with Deep Reinforcement Learning for URLLC and eMBB Services

Abderrahime Filali, Zoubeir Mlika+2 more|arXiv (Cornell University)|Feb 13, 2022

Software-Defined Networks and 5G4 citations

TL;DR

This paper proposes a two-timescale SDN-based RAN slicing framework that dynamically allocates radio resources to URLLC and eMBB services using deep reinforcement learning. In the large timescale, an SDN controller allocates resource blocks (RBs) to gNodeBs via an EXP3-based single-agent MDP; in the short timescale, gNodeBs use multi-agent deep Q-learning (DQL) to schedule RBs to users and borrow from neighbors when needed, significantly improving QoS for both services under dynamic traffic.

ABSTRACT

Radio access network (RAN) slicing is a key technology that enables 5G network to support heterogeneous requirements of generic services, namely ultra-reliable low-latency communication (URLLC) and enhanced mobile broadband (eMBB). In this paper, we propose a two time-scales RAN slicing mechanism to optimize the performance of URLLC and eMBB services. In a large time-scale, an SDN controller allocates radio resources to gNodeBs according to the requirements of the eMBB and URLLC services. In a short time-scale, each gNodeB allocates its available resources to its end-users and requests, if needed, additional resources from adjacent gNodeBs. We formulate this problem as a non-linear binary program and prove its NP-hardness. Next, for each time-scale, we model the problem as a Markov decision process (MDP), where the large-time scale is modeled as a single agent MDP whereas the shorter time-scale is modeled as a multi-agent MDP. We leverage the exponential-weight algorithm for exploration and exploitation (EXP3) to solve the single-agent MDP of the large time-scale MDP and the multi-agent deep Q-learning (DQL) algorithm to solve the multi-agent MDP of the short time-scale resource allocation. Extensive simulations show that our approach is efficient under different network parameters configuration and it outperforms recent benchmark solutions.

Motivation & Objective

• To address the high signaling overhead and inflexible resource allocation in centralized and multi-level RAN slicing approaches.
• To enable dynamic, scalable, and QoS-aware radio resource allocation for heterogeneous 5G services (URLLC and eMBB) in SDN-enabled RANs.
• To reduce dependency on frequent SDN controller interactions by decentralizing short-timescale resource scheduling at gNodeBs.
• To ensure reliable and low-latency support for URLLC and high-throughput for eMBB services under dynamic network conditions.
• To develop a robust, adaptive resource allocation mechanism that can handle fluctuating user demands and limited pre-allocated resources.

Proposed method

• Formulates the radio resource block (RB) allocation problem as a non-linear binary program and proves its NP-hardness.
• Models the large-timescale RB allocation by the SDN controller as a single-agent Markov Decision Process (MDP), solved using the EXP3 algorithm for exploration and exploitation.
• Models the short-timescale gNodeB-level resource allocation as a multi-agent MDP, solved using multi-agent deep Q-learning (DQL) with double DQN and experience replay for stability and performance.
• Enables gNodeBs to dynamically borrow unused RBs from neighboring gNodeBs when pre-allocated resources are insufficient, improving resource utilization and QoS.
• Integrates robust deep reinforcement learning components, including target networks and replay memory, to enhance learning convergence and decision quality.
• Uses a two-tier architecture: long-term RB allocation by SDN controller to gNodeBs, followed by local, fast scheduling at each gNodeB based on real-time user demands.

Figure 1 : Resource block allocation procedure

Experimental results

Research questions

• RQ1 How can SDN-based RAN slicing efficiently balance resource allocation between URLLC and eMBB services under dynamic traffic conditions?
• RQ2 What is the optimal trade-off between signaling overhead and QoS satisfaction in hierarchical RAN slicing architectures?
• RQ3 Can a multi-agent deep reinforcement learning approach outperform centralized or single-agent methods in local gNodeB-level resource scheduling?
• RQ4 How does inter-gNodeB RB borrowing improve resource availability and service reliability during sudden traffic spikes?
• RQ5 To what extent does the proposed two-timescale mechanism reduce dependency on frequent SDN controller interactions while maintaining QoS for latency-sensitive and high-throughput services?

Key findings

• SAMA-RL (the proposed method) consistently outperforms 3-SRA and 1-SRA in maximizing total sum data rate across all tested minimum data rate thresholds.
• The performance gap between SAMA-RL and benchmarks increases with higher minimum data rate thresholds, demonstrating the method’s robustness under stringent QoS constraints.
• SAMA-RL achieves a higher number of end-users meeting the minimum data rate threshold compared to 3-SRA and 1-SRA across all configurations.
• SAMA-RL significantly outperforms 3-SRA and 1-SRA in satisfying end-user delay requirements, especially under high delay thresholds.
• The multi-agent DQL-based borrowing mechanism enables gNodeBs to dynamically access additional RBs, reducing service blocking and improving QoS resilience.
• Simulation results confirm that the proposed two-timescale mechanism reduces signaling overhead while maintaining or improving QoS for both URLLC and eMBB services.

Figure 2 : Single-agent multi-agent interaction with the MDP environment.