[Paper Review] Learning from Peers: Transfer Reinforcement Learning for Joint Radio and Cache Resource Allocation in 5G Network Slicing.

Hao Zhou, Melike Erol‐Kantarci+1 more|arXiv (Cornell University)|Sep 16, 2021

Software-Defined Networks and 5G30 references4 citations

TL;DR

This paper proposes a transfer reinforcement learning (TRL) framework for joint radio and cache resource allocation in 5G network slicing, leveraging expert agents to accelerate learning and improve performance. The QTRL and ASTRl algorithms achieve 41.6% higher eMBB throughput and 40.3% lower URLLC delay compared to PPF-TTL, with faster convergence than Q-learning.

ABSTRACT

Radio access network (RAN) slicing is an important part of network slicing in 5G. The evolving network architecture requires the orchestration of multiple network resources such as radio and cache resources. In recent years, machine learning (ML) techniques have been widely applied for network slicing. However, most existing works do not take advantage of the knowledge transfer capability in ML. In this paper, we propose a transfer reinforcement learning (TRL) scheme for joint radio and cache resources allocation to serve 5G RAN slicing.We first define a hierarchical architecture for the joint resources allocation. Then we propose two TRL algorithms: Q-value transfer reinforcement learning (QTRL) and action selection transfer reinforcement learning (ASTRL). In the proposed schemes, learner agents utilize the expert agents' knowledge to improve their performance on target tasks. The proposed algorithms are compared with both the model-free Q-learning and the model-based priority proportional fairness and time-to-live (PPF-TTL) algorithms. Compared with Q-learning, QTRL and ASTRL present 23.9% lower delay for Ultra Reliable Low Latency Communications slice and 41.6% higher throughput for enhanced Mobile Broad Band slice, while achieving significantly faster convergence than Q-learning. Moreover, 40.3% lower URLLC delay and almost twice eMBB throughput are observed with respect to PPF-TTL.

Motivation & Objective

• To address the challenge of slow convergence and suboptimal resource allocation in 5G radio access network slicing.
• To exploit knowledge transfer from expert agents to improve learning efficiency and performance in joint radio and cache resource allocation.
• To design a hierarchical architecture for coordinated resource orchestration across multiple network slices.
• To outperform existing model-free and model-based approaches in terms of delay, throughput, and convergence speed.

Proposed method

• Propose a hierarchical architecture for joint radio and cache resource allocation in 5G RAN slicing.
• Develop Q-value transfer reinforcement learning (QTRL), where learner agents transfer Q-value functions from expert agents.
• Design action selection transfer reinforcement learning (ASTRL), where learners transfer action selection policies from experts.
• Train expert agents on source tasks to generate reusable knowledge for target tasks.
• Apply transfer learning to accelerate convergence and improve performance on target resource allocation tasks.
• Compare the proposed TRL schemes with Q-learning (model-free) and PPF-TTL (model-based) baselines.

Experimental results

Research questions

• RQ1 Can transfer reinforcement learning improve convergence speed and performance in joint radio and cache resource allocation for 5G network slicing?
• RQ2 How does knowledge transfer from expert agents affect the performance of learner agents in URLLC and eMBB slices?
• RQ3 To what extent do QTRL and ASTRl outperform Q-learning and PPF-TTL in terms of delay and throughput?
• RQ4 What is the impact of hierarchical resource allocation architecture on system performance in multi-slice 5G networks?

Key findings

• QTRL and ASTRl reduce URLLC slice delay by 23.9% compared to Q-learning.
• ASTRL achieves 41.6% higher throughput in the eMBB slice compared to Q-learning.
• The proposed TRL schemes converge significantly faster than Q-learning due to knowledge transfer.
• QTRL and ASTRl reduce URLLC delay by 40.3% compared to the PPF-TTL baseline.
• ASTRL nearly doubles the eMBB throughput relative to PPF-TTL.
• The TRL-based approaches outperform both model-free and model-based baselines in key performance metrics across all slices.