[论文解读] Average Reward Adjusted Discounted Reinforcement Learning: Near-Blackwell-Optimal Policies for Real-World Applications

Manuel Schneckenreither|arXiv (Cornell University)|Apr 2, 2020

Reinforcement Learning in Robotics参考文献 38被引用 6

一句话总结

本文提出了一种新型算法——平均奖励修正折扣强化学习（ARA-DRL），该算法将平均奖励估计与折扣价值学习分离，以在具有持续非零奖励的非周期性、现实世界运筹学问题中实现近Blackwell最优策略。与标准折扣强化学习不同，ARA-DRL通过使用折扣价值的洛朗级数展开，防止了价值函数错位。在M/M/1准入控制问题上，ARA-DRL在稳定平均奖励下持续推断出最优策略，表现优于Q-learning和标准强化学习。

ABSTRACT

Although in recent years reinforcement learning has become very popular the number of successful applications to different kinds of operations research problems is rather scarce. Reinforcement learning is based on the well-studied dynamic programming technique and thus also aims at finding the best stationary policy for a given Markov Decision Process, but in contrast does not require any model knowledge. The policy is assessed solely on consecutive states (or state-action pairs), which are observed while an agent explores the solution space. The contributions of this paper are manifold. First we provide deep theoretical insights to the widely applied standard discounted reinforcement learning framework, which give rise to the understanding of why these algorithms are inappropriate when permanently provided with non-zero rewards, such as costs or profit. Second, we establish a novel near-Blackwell-optimal reinforcement learning algorithm. In contrary to former method it assesses the average reward per step separately and thus prevents the incautious combination of different types of state values. Thereby, the Laurent Series expansion of the discounted state values forms the foundation for this development and also provides the connection between the two approaches. Finally, we prove the viability of our algorithm on a challenging problem set, which includes a well-studied M/M/1 admission control queuing system. In contrast to standard discounted reinforcement learning our algorithm infers the optimal policy on all tested problems. The insights are that in the operations research domain machine learning techniques have to be adapted and advanced to successfully apply these methods in our settings.

研究动机与目标

• 识别标准折扣强化学习为何在具有持续非零奖励（如成本或利润）的真实世界运筹学问题中失效。
• 开发一种在平均奖励与折扣价值分离理论上具有坚实基础的强化学习算法，以避免价值函数错位。
• 在具有挑战性的非周期性决策问题（包括M/M/1排队系统）上实证验证所提方法。
• 证明ARA-DRL在标准强化学习失效的场景下（尤其是在非零平均奖励条件下）可实现近Blackwell最优策略。
• 提供一种理论可靠、计算高效的强化学习框架，可适用于分层供应链与生产计划系统。

提出的方法

• 该方法提出一种新型算法——平均奖励修正折扣强化学习（ARA-DRL），将平均奖励估计与折扣价值学习解耦。
• 利用折扣状态值的洛朗级数展开，从瞬态折扣价值分量中解析分离出平均奖励分量。
• ARA-DRL维持两个独立的学习过程：一个用于估计长期平均奖励，另一个用于更新折扣状态-动作值。
• 该算法应用一种改进的更新规则，基于估计的平均奖励调整折扣价值函数，防止错误的价值传播。
• 采用衰减学习率和自适应ε的ε-greedy探索策略，以稳定学习和策略收敛。
• 该方法以表格型Q-learning风格的更新方式实现，并在具有已知最优策略的经典M/M/1准入控制问题上进行了验证。

实验结果

研究问题

• RQ1 为何标准折扣强化学习在具有持续非零奖励（如利润或成本函数）的运筹学问题中会失效？
• RQ2 能否设计一种强化学习算法，通过显式建模平均奖励分量，实现近Blackwell最优性？
• RQ3 在非周期性环境中，将平均奖励估计与折扣价值学习分离如何提升策略的稳定性和收敛性？
• RQ4 在现实世界运筹学问题中，ARA-DRL在奖励累积和策略准确性方面相较于标准Q-learning和折扣强化学习能提升多少？
• RQ5 平均奖励估计的稳定性在ARA-DRL算法的收敛性和性能中起到何种作用？

主要发现

• 当γ₁ = 1.0和γ₁ = 0.999时，ARA-DRL成功推断出M/M/1准入控制系统中接纳3个任务的Blackwell最优策略，每步平均奖励分别为29.965和30.272。
• ARA-DRL在γ₁ = 0.999和γ₁ = 1.0时实现了1.12的平均队列长度，与最优解完全一致，证实了策略最优性。
• 标准Q-learning在γ₁ = 0.99、0.999和0.5时均未能找到接近最优的策略，平均奖励低于33，平均队列长度低于0.2。
• 即使采用ε敏感的动作选择，Q-learning也未能达到接纳3个任务的Blackwell最优策略，凸显其在非零平均奖励设定下的局限性。
• 对ARA-DRL的超参数调优（如将学习率衰减提高至0.8）增强了稳定性，并在保持最优策略推断的同时实现了ε ≤ 1。
• Friedman检验在平均奖励和平均队列长度上均拒绝了原假设（p < 3.399e-32 和 p < 2.223e-35），证实了ARA-DRL优越性的统计显著性。