[论文解读] Design and Comparison of Reward Functions in Reinforcement Learning for Energy Management of Sensor Nodes

Yohann Rioual, Yannick Le Moullec+3 more|arXiv (Cornell University)|Jun 2, 2021

Reinforcement Learning in Robotics参考文献 23被引用 5

一句话总结

本文设计并评估了七种基于Q-learning的无线传感器节点能量管理奖励函数，表明具有动态平衡参数的自适应奖励函数（R6和R7）通过根据电池荷电状态优化性能与能耗之间的权衡，显著提升了能量效率并减少了学习时间。

ABSTRACT

Interest in remote monitoring has grown thanks to recent advancements in Internet-of-Things (IoT) paradigms. New applications have emerged, using small devices called sensor nodes capable of collecting data from the environment and processing it. However, more and more data are processed and transmitted with longer operational periods. At the same, the battery technologies have not improved fast enough to cope with these increasing needs. This makes the energy consumption issue increasingly challenging and thus, miniaturized energy harvesting devices have emerged to complement traditional energy sources. Nevertheless, the harvested energy fluctuates significantly during the node operation, increasing uncertainty in actually available energy resources. Recently, approaches in energy management have been developed, in particular using reinforcement learning approaches. However, in reinforcement learning, the algorithm's performance relies greatly on the reward function. In this paper, we present two contributions. First, we explore five different reward functions to identify the most suitable variables to use in such functions to obtain the desired behaviour. Experiments were conducted using the Q-learning algorithm to adjust the energy consumption depending on the energy harvested. Results with the five reward functions illustrate how the choice thereof impacts the energy consumption of the node. Secondly, we propose two additional reward functions able to find the compromise between energy consumption and a node performance using a non-fixed balancing parameter. Our simulation results show that the proposed reward functions adjust the node's performance depending on the battery level and reduce the learning time.

研究动机与目标

• 确定面向能量采集型传感器节点能量管理的最优奖励函数组件。
• 评估不同奖励函数设计对Q-learning在管理能量消耗方面性能的影响。
• 开发基于电池荷电状态动态平衡能量消耗与节点性能的自适应奖励函数。
• 在保持或提升能量效率的同时，减少传感器节点运行的学习时间。

提出的方法

• 设计了五种基线奖励函数（R1–R5），其组合使用了电池荷电状态、采集的能量、测量频率和性能指标。
• 提出了两种高级奖励函数（R6和R7），其平衡参数不固定，可根据电池电量进行缩放，以优化性能与能耗之间的权衡。
• 采用Q-learning作为强化学习算法，在不同能量采集条件下模拟环境中训练智能体。
• 通过多日模拟评估奖励函数，将电池充电水平和测量频率作为关键性能指标进行跟踪。
• R6和R7的设计融入了基于当前电池状态动态调整的平衡参数，提升了适应能力。
• 通过延长模拟（24天运行）验证了Q值的收敛性，确认了策略学习的稳定性。

实验结果

研究问题

• RQ1 不同奖励函数组件（如电池电量、采集能量、测量频率）如何影响Q-learning在能量管理中的学习与性能？
• RQ2 固定参数与动态平衡参数对能耗与节点性能之间权衡的影响是什么？
• RQ3 能够根据电池荷电状态自适应调整的奖励函数是否可提升学习效率与能量管理效果？
• RQ4 哪种奖励函数设计能使能量采集型传感器节点实现最快收敛与最稳定行为？

主要发现

• 奖励函数R1和R2包含平衡参数β，能有效将能耗调整至与采集能量水平相匹配，表现出优异性能。
• 函数R5仅基于主导运动频率，虽能适应能量采集，但在需要时未能优先考虑性能。
• 函数R3和R4表现欠佳：R3缺乏能耗与能量采集之间的关联，而R4在能量较低时未能降低能耗。
• R6和R7因采用动态平衡参数，优于固定参数函数，能根据电池状态自适应调整，从而减少学习时间。
• 模拟结果证实，Q值在24天内实现收敛，表明在不同能量条件下具备稳定且自适应的行为。
• 所提出的奖励函数（R6和R7）在不同电池容量下均表现出鲁棒性，且无需预先了解系统组件。