[论文解读] When Multiple Agents Learn to Schedule: A Distributed Radio Resource Management Framework

Navid Naderializadeh, Jaroslaw J. Sydir+3 more|arXiv (Cornell University)|Jun 20, 2019

Advanced MIMO Systems Optimization参考文献 15被引用 4

一句话总结

该论文提出了一种用于密集无线网络中分布式无线资源管理的多智能体深度强化学习（DRL）框架，其中每个接入点（AP）作为DQN智能体，基于本地和延迟的邻近观测做出链路调度决策。该框架在平均用户吞吐量与第5百分位用户吞吐量之间实现了出色的权衡，优于去中心化基线方法，并接近集中式穷举搜索的性能，同时在网络密度变化时表现出强健性。

ABSTRACT

Interference among concurrent transmissions in a wireless network is a key factor limiting the system performance. One way to alleviate this problem is to manage the radio resources in order to maximize either the average or the worst-case performance. However, joint consideration of both metrics is often neglected as they are competing in nature. In this article, a mechanism for radio resource management using multi-agent deep reinforcement learning (RL) is proposed, which strikes the right trade-off between maximizing the average and the $5^{th}$ percentile user throughput. Each transmitter in the network is equipped with a deep RL agent, receiving partial observations from the network (e.g., channel quality, interference level, etc.) and deciding whether to be active or inactive at each scheduling interval for given radio resources, a process referred to as link scheduling. Based on the actions of all agents, the network emits a reward to the agents, indicating how good their joint decisions were. The proposed framework enables the agents to make decisions in a distributed manner, and the reward is designed in such a way that the agents strive to guarantee a minimum performance, leading to a fair resource allocation among all users across the network. Simulation results demonstrate the superiority of our approach compared to decentralized baselines in terms of average and $5^{th}$ percentile user throughput, while achieving performance close to that of a centralized exhaustive search approach. Moreover, the proposed framework is robust to mismatches between training and testing scenarios. In particular, it is shown that an agent trained on a network with low transmitter density maintains its performance and outperforms the baselines when deployed in a network with a higher transmitter density.

研究动机与目标

• 解决超密集无线网络中的干扰和资源分配不公平问题。
• 设计一种分布式无线资源管理机制，公平地平衡平均吞吐量与最小用户吞吐量。
• 开发一种可扩展且鲁棒的学习型调度框架，能够抵御训练与部署环境之间的不匹配。
• 通过混合集中式训练与分布式推理的架构，实现DRL智能体在真实网络中的实际部署。

提出的方法

• 每个接入点（AP）配备一个深度Q网络（DQN）智能体，基于对本地信道质量与干扰水平的部分观测做出调度决策。
• 智能体接收来自邻近AP的延迟且不频繁的观测，以反映现实世界中的通信约束。
• 采用集中式训练框架，聚合所有智能体的经验，并使用一种奖励函数，优先考虑高平均吞吐量与最低的第5百分位用户速率，以确保公平性。
• 训练过程使用回放缓冲区，并由中心实体定期更新权重，以稳定学习并提升收敛性。
• 该框架采用三级通信架构：实时推理链路、中频次的训练数据收集，以及低频次的策略更新与暂停/终止信号。
• 系统设计使得单一训练好的策略能够泛化至不同网络密度而无需重新训练。

实验结果

研究问题

• RQ1 多智能体DRL框架是否能在干扰受限的无线网络中实现平均吞吐量与最小用户吞吐量之间的平衡权衡？
• RQ2 在低密度网络中训练的DRL智能体在更高密度部署场景中是否具有良好泛化能力？
• RQ3 集中式训练与分布式推理方法在动态且异构的网络条件下是否能保持性能与公平性？
• RQ4 在真实无线网络中部署基于DRL的调度存在哪些实际挑战，以及如何缓解这些问题？

主要发现

• 所提出的DRL框架在平均吞吐量与第5百分位用户吞吐量方面均优于全复用与时分复用（TDM）基线方法。
• 该框架的性能接近集中式穷举搜索，表明其在资源分配方面接近最优效率。
• 在四AP网络中训练的智能体在部署至最多含十AP的网络时仍保持强劲性能，表明其对网络密度变化具有高度鲁棒性。
• 该框架对分布偏移具有鲁棒性，在测试场景与训练条件在发射机密度与信道条件上存在差异时，仍能保持有效性能。
• 使用延迟且不频繁的邻近观测使得实际部署成为可能，同时保持了学习的有效性。