[论文解读] Reward Biased Maximum Likelihood Estimation for Reinforcement Learning

Akshay Mete, Rahul Singh+2 more|arXiv (Cornell University)|Nov 16, 2020

Advanced Bandit Algorithms Research参考文献 37被引用 6

一句话总结

该论文针对未知马尔可夫决策过程中的强化学习问题，提出了奖励偏差最大似然估计（RBMLE）方法，通过偏好最优奖励更高的参数来平衡探索与利用。该方法证明在 $T$ 步内可实现 $O(\log T)$ 的遗憾，与当前最先进算法性能相当，并在模拟实验中表现出优于 UCRL2 和 Thompson Sampling 的实际性能。

ABSTRACT

The Reward-Biased Maximum Likelihood Estimate (RBMLE) for adaptive control of Markov chains was proposed to overcome the central obstacle of what is variously called the fundamental "closed-identifiability problem" of adaptive control, the "dual control problem", or, contemporaneously, the "exploration vs. exploitation problem". It exploited the key observation that since the maximum likelihood parameter estimator can asymptotically identify the closed-transition probabilities under a certainty equivalent approach, the limiting parameter estimates must necessarily have an optimal reward that is less than the optimal reward attainable for the true but unknown system. Hence it proposed a counteracting reverse bias in favor of parameters with larger optimal rewards, providing a solution to the fundamental problem alluded to above. It thereby proposed an optimistic approach of favoring parameters with larger optimal rewards, now known as "optimism in the face of uncertainty". The RBMLE approach has been proved to be long-term average reward optimal in a variety of contexts. However, modern attention is focused on the much finer notion of "regret", or finite-time performance. Recent analysis of RBMLE for multi-armed stochastic bandits and linear contextual bandits has shown that it not only has state-of-the-art regret, but it also exhibits empirical performance comparable to or better than the best current contenders, and leads to strikingly simple index policies. Motivated by this, we examine the finite-time performance of RBMLE for reinforcement learning tasks that involve the general problem of optimal control of unknown Markov Decision Processes. We show that it has a regret of $\mathcal{O}( \log T)$ over a time horizon of $T$ steps, similar to state-of-the-art algorithms. Simulation studies show that RBMLE outperforms other algorithms such as UCRL2 and Thompson Sampling.

研究动机与目标

• 解决未知马尔可夫决策过程中的探索-利用权衡问题。
• 开发一种在模型不确定性下兼顾学习与性能的有限时间遗憾最优算法。
• 将此前用于自适应控制和多臂赌博机的 RBMLE 框架扩展至具有平均奖励准则的一般 MDP。
• 证明 RBMLE 可实现当前最先进水平的遗憾边界，并在强化学习设置中展现出优越的实际性能。

提出的方法

• RBMLE 在最大似然估计过程中引入奖励偏差，偏好最优奖励更高的参数估计。
• 基于似然性构建转移模型估计，随后施加反向偏差，以偏好具有更高潜在长期奖励的模型。
• 该方法采用确定性等价方法，但对参数估计中的偏差进行校正，以避免导致次优策略。
• 算法根据偏差参数分布下最优策略的乐观估计选择动作。
• 利用标准 MLE 在渐近意义上可识别真实转移模型的理论性质，但会低估最优奖励，因此施加反向偏差。
• 遗憾分析基于平均奖励准则，性能衡量为最优与实际累积奖励随时间的差值。

实验结果

研究问题

• RQ1 RBMLE 是否能在未知 MDP 的平均奖励设置下实现 $O(\log T)$ 的遗憾？
• RQ2 RBMLE 在有限时间内的性能与 UCRL2 和 Thompson Sampling 等最先进算法相比如何？
• RQ3 奖励偏差机制是否能有效平衡一般 MDP 中的探索与利用？
• RQ4 RBMLE 是否可从赌博机设置扩展至完整 MDP，同时保持遗憾最优性？

主要发现

• RBMLE 在 $T$ 步的时间范围内实现了 $O(\log T)$ 的遗憾边界，与最先进算法性能相当。
• 模拟结果表明，RBMLE 在累积奖励和收敛速度方面优于 UCRL2 和 Thompson Sampling。
• 该算法在各种强化学习任务中表现出与当前最佳方法相当或更优的实际性能。
• 奖励偏差最大似然估计方法通过偏好具有更高潜在奖励的乐观参数估计，成功解决了探索-利用困境。