[论文解读] Intelligent Power Control for Spectrum Sharing: A Deep Reinforcement Learning Approach.

Jun Fang, Xingjian Li+3 more|arXiv (Cornell University)|Dec 20, 2017

Cognitive Radio Networks and Spectrum Sensing参考文献 35被引用 10

一句话总结

本文提出了一种基于深度强化学习（DRL）的功率控制框架，用于认知无线电网络中的次用户，使其能够在不了解主用户行为先验知识的情况下，智能地调整发射功率。通过利用空间部署的传感器节点监测信号强度，DRL智能体在少数几次迭代内即可学习实现两个用户的同时成功传输，确保服务质量（QoS）要求得到满足。

ABSTRACT

We consider the problem of spectrum sharing in a cognitive radio system consisting of a primary user and a secondary user. The primary user and the secondary user work in a non-cooperative manner, and independently adjust their respective transmit power. Specifically, the primary user is assumed to update its transmit power based on a pre-defined power control policy. The secondary user does not have any knowledge about the primary user's transmit power, neither its power control strategy. The objective of this paper is to develop a learning-based power control method for the secondary user in order to share the common spectrum with the primary user. To assist the secondary user, a set of sensor nodes are spatially deployed to collect the received signal strength information at different locations in the wireless environment. We develop a deep reinforcement learning-based method for the secondary user, based on which the secondary user can intelligently adjust its transmit power such that after a few rounds of interaction with the primary user, both the primary user and the secondary user can transmit their own data successfully with required qualities of service. Our experimental results show that the secondary user can interact with the primary user efficiently to reach a goal state (defined as a state in which both the primary and the secondary users can successfully transmit their own data) from any initial states within a few number of iterations.

研究动机与目标

• 解决认知无线电网络中次用户缺乏对主用户功率控制策略了解时的频谱共享挑战。
• 使次用户能够在非协作环境中动态调整发射功率，以维持服务质量。
• 开发一种基于学习的方法，使次用户通过与主用户的交互，收敛至成功的传输状态。
• 利用分布式传感器节点收集实时信号强度信息，以支持明智的决策制定。
• 在动态无线环境中实现高效频谱接入，同时最小化干扰并保证高可靠性。

提出的方法

• 训练一个深度Q网络（DQN）强化学习智能体，基于传感器节点的观测结果来优化次用户的发射功率。
• 传感器节点在空间中分布部署，用于收集无线环境中的接收信号强度（RSS）测量值。
• DRL智能体的状态空间包括来自多个传感器的RSS读数，用以表示当前的无线电环境。
• 动作空间由次用户可选择的离散功率级别组成，用于数据传输。
• 奖励函数被设计为鼓励主用户和次用户均成功传输，同时最小化干扰。
• DRL智能体通过与主用户的试错式交互进行学习，持续更新其策略以最大化长期奖励。

实验结果

研究问题

• RQ1 次用户是否能够在不了解主用户功率控制策略的情况下，有效学习调整其发射功率？
• RQ2 基于DRL的智能体需要多快才能收敛至主用户和次用户均能成功传输且满足所需QoS的状态？
• RQ3 分布式传感器节点在实现精确且及时的功率控制决策中发挥何种作用？
• RQ4 该DRL方法在不同初始状态和环境条件下具有多强的鲁棒性？
• RQ5 传感器部署位置和RSS测量质量对次用户学习性能有何影响？

主要发现

• 次用户可从任意初始状态在少数几次迭代内成功达到目标状态——即主用户和次用户均成功传输。
• 基于DRL的方法使频谱共享更加高效，且无需明确知晓主用户的功率控制策略。
• 使用空间分布的传感器节点显著提高了DRL智能体对环境状态估计的准确性。
• 所提方法在各种初始条件下均表现出可靠性能，展现出强大的泛化能力。
• 学习过程收敛迅速，表明该方法在动态认知无线电环境中具备实时部署的可行性。