[论文解读] Learning from Peers: Transfer Reinforcement Learning for Joint Radio and Cache Resource Allocation in 5G Network Slicing.

Hao Zhou, Melike Erol‐Kantarci+1 more|arXiv (Cornell University)|Sep 16, 2021

Software-Defined Networks and 5G参考文献 30被引用 4

一句话总结

本文提出了一种用于5G网络切片中联合无线与缓存资源分配的迁移强化学习（TRL）框架，利用专家智能体加速学习并提升性能。QTRL和ASTRL算法相较于PPF-TTL，实现eMBB吞吐量提升41.6%、URLLC延迟降低40.3%，且收敛速度优于Q-learning。

ABSTRACT

Radio access network (RAN) slicing is an important part of network slicing in 5G. The evolving network architecture requires the orchestration of multiple network resources such as radio and cache resources. In recent years, machine learning (ML) techniques have been widely applied for network slicing. However, most existing works do not take advantage of the knowledge transfer capability in ML. In this paper, we propose a transfer reinforcement learning (TRL) scheme for joint radio and cache resources allocation to serve 5G RAN slicing.We first define a hierarchical architecture for the joint resources allocation. Then we propose two TRL algorithms: Q-value transfer reinforcement learning (QTRL) and action selection transfer reinforcement learning (ASTRL). In the proposed schemes, learner agents utilize the expert agents' knowledge to improve their performance on target tasks. The proposed algorithms are compared with both the model-free Q-learning and the model-based priority proportional fairness and time-to-live (PPF-TTL) algorithms. Compared with Q-learning, QTRL and ASTRL present 23.9% lower delay for Ultra Reliable Low Latency Communications slice and 41.6% higher throughput for enhanced Mobile Broad Band slice, while achieving significantly faster convergence than Q-learning. Moreover, 40.3% lower URLLC delay and almost twice eMBB throughput are observed with respect to PPF-TTL.

研究动机与目标

• 为解决5G无线接入网络切片中收敛速度慢与资源分配次优的挑战。
• 利用专家智能体的知识迁移，提升联合无线与缓存资源分配中的学习效率与性能。
• 设计一种分层架构，实现多个网络切片间协调的资源编排。
• 在延迟、吞吐量与收敛速度方面，超越现有的无模型与有模型方法。

提出的方法

• 提出一种用于5G RAN切片中联合无线与缓存资源分配的分层架构。
• 开发Q值迁移强化学习（QTRL），学习智能体从专家智能体迁移Q值函数。
• 设计动作选择迁移强化学习（ASTRL），学习智能体从专家智能体迁移动作选择策略。
• 在源任务上训练专家智能体，生成可用于目标任务的可重用知识。
• 应用迁移学习以加速目标资源分配任务的收敛并提升性能。
• 将所提出的TRL方案与Q-learning（无模型）和PPF-TTL（有模型）基线方法进行比较。

实验结果

研究问题

• RQ1 迁移强化学习能否提升5G网络切片中联合无线与缓存资源分配的收敛速度与性能？
• RQ2 来自专家智能体的知识迁移对URLLC与eMBB切片中学习智能体的性能有何影响？
• RQ3 QTRL与ASTRL在延迟与吞吐量方面相较于Q-learning与PPF-TTL的优越程度如何？
• RQ4 分层资源分配架构对多切片5G网络系统性能有何影响？

主要发现

• QTRL与ASTRL相较于Q-learning，将URLLC切片延迟降低了23.9%。
• ASTRL相较于Q-learning，使eMBB切片的吞吐量提升了41.6%。
• 由于知识迁移，所提出的TRL方案相较于Q-learning实现显著更快的收敛速度。
• QTRL与ASTRL相较于PPF-TTL基线，将URLLC延迟降低了40.3%。
• ASTRL相较于PPF-TTL，几乎使eMBB吞吐量翻倍。
• 基于TRL的方法在所有切片的关键性能指标上，均优于无模型与有模型基线方法。