[论文解读] Online Off-policy Prediction

Sina Ghiassian|arXiv (Cornell University)|Nov 6, 2018

Advanced Bandit Algorithms Research参考文献 26被引用 14

一句话总结

本文对在线离策略时序差分学习算法在值函数预测中的表现进行了全面的实证研究，评估了Emphatic-TD、Gradient-TD和V-trace等方法在两个具有挑战性的微型世界中的表现。研究发现，ETD(λ)在渐近性能和超参数鲁棒性方面表现更优，而使用重要性采样对整个TD误差进行校正始终优于仅对值函数估计进行部分校正。

ABSTRACT

In this dissertation, we study online off-policy temporal-difference learning algorithms, a class of reinforcement learning algorithms that can learn predictions in an efficient and scalable manner. The contributions of this dissertation are one of the two kinds: (1) empirically studying existing off-policy learning algorithms, or, (2) exploring new algorithmic ideas. In reinforcement learning, it is not uncommon for an agent to learn about one policy, called the target policy, while behaving using a different policy, called the behavior policy. When the target and behavior policies are different, learning is `off' the policy in the sense that the data is from a different source than the target policy. Our first contribution is a novel Gradient-TD algorithm called TDRC. Gradient-TD algorithms are one of the most important families of off-policy learning algorithms due to their favorable convergence guarantees. Previous research showed that Gradient-TD algorithms learn slower than some other unsound algorithms such as Off-policy TD (Maei, 2011). In addition, Gradient-TD algorithms are not as easy to use as Off-policy TD because they have two learned weight vectors and two step-size parameters. Our contribution is a new algorithm called temporal-difference learning with regularized corrections (TDRC), that behaves as well as Off-policy TD, when Off-policy TD performs well but is sound in cases where Off-policy TD diverges. TDRC provides a standard way of setting the second step-size parameter and is easier to use than other Gradient-TD algorithms. We empirically investigate the performance of TDRC across a range of problems, for both prediction and control, and for both linear and non-linear function approximation, and show that it consistently outperforms other Gradient-TD algorithms. We derive the control variant of TDRC and show that it could be a better alternative to existing algorithms for solving complex control problems with neural networks. One of the important contributions of this thesis is a comprehensive empirical study of off-policy prediction learning algorithms. The study is one of the most thorough studies of off-policy prediction learning algorithms to date in terms of the number of algorithms it includes and is unique in that the performance of algorithms with respect to each of the algorithms parameters is assessed individually. We present empirical results with eleven prominent off-policy learning algorithms that use linear function approximation: five Gradient-TD methods, two Emphatic-TD methods, Off-policy TD, V-trace, Tree Backup, and ABQ. We assess the performance of the algorithms in terms of their learning speed, asymptotic error, and sensitivity to the step-size and bootstrapping parameters on a simple problem, called the Collision task, that has eight states and two actions. On the Collision task, we found that the two Emphatic-TD algorithms learned the fastest, reached the lowest errors, and were robust to parameter settings. V-trace, Tree Backup, and ABQ were no faster and had higher asymptotic error than other algorithms. Another one of the main contributions of this thesis is the first empirical study of off-policy prediction learning algorithms with a focus on one of the most important challenges in off-policy learning: slow learning. When learning off-policy, importance sampling ratios are used to correct for the differences between the target and behavior policies. These ratios can be large and of high variance. The step-size parameter should be set small to control this high variance, which in turn slows down learning. In this study, we found that the algorithms' performance is highly affected by the variance induced by the importance sampling ratios. Data shows that algorithms that adapt the bootstrapping parameter during learning, such as V-trace are less affected by the high variance than the other algorithms. We observed that Emphatic TD tends to have lower asymptotic error than other algorithms, but exhibits high variance and does not learn well when the product of importance sampling ratios is large. The final contribution of this thesis is a step-size adaptation algorithm, called Step-size Ratchet, that when combined with Emphatic TD, significantly improves its learning speed. The proposed algorithm keeps the step-size parameter as large as possible and only ratchets it down when necessary. We establish the effectiveness of the combination of Emphatic TD and Step-size Ratchet by comparing it with the combination of Emphatic TD and other step-size adaptation algorithms on tasks where learning fast is challenging. (Abridged abstract)

研究动机与目标

• 为填补现代在线离策略预测算法相对性能理解方面的实证空白。
• 评估重要性采样位置的影响——即对整个TD误差进行校正与仅对值函数估计进行校正——对学习稳定性和性能的影响。
• 评估关键算法在不同超参数和环境方差下的敏感性、收敛速度和渐近性能。
• 明确Emphatic-TD、Gradient-TD、V-trace和Tree Backup等方法在固定基函数设置下的实际优势与局限性。

提出的方法

• 作者在两个具有固定基函数表示的复杂微型世界中，评估了14种在线离策略预测算法，包括Emphatic-TD、Gradient-TD、V-trace、Tree Backup和ABTD(ζ)。
• 系统性地改变学习率、λ和β等超参数，以评估其敏感性和收敛行为。
• 重要性采样校正应用于整个TD误差或仅应用于值函数估计，其对性能的影响通过实证比较进行评估。
• 研究采用曲线下面积（AUC）优化来调整超参数，并评估学习速度、方差和最终性能。
• 分析包括重要性采样比的低方差和高方差两种情形，以评估鲁棒性。
• 作者对比了具有和不具有显式先验校正（如V-trace）的方法，以及不同资格迹机制的表现。

实验结果

研究问题

• RQ1 不同离策略预测算法在渐近性能、收敛速度和超参数敏感性方面如何比较？
• RQ2 使用重要性采样对整个TD误差项进行校正是否优于仅对值函数估计进行校正？
• RQ3 在重要性采样比高方差和长资格迹条件下，ETD(λ)和V-trace等方法表现如何？
• RQ4 先验校正（如V-trace和Tree Backup中所采用）在稳定学习和提升性能方面起到什么作用？
• RQ5 Gradient-TD的混合或近端变体在实践中是否相比标准Gradient-TD有显著改进？

主要发现

• Emphatic-TD(λ)在所有测试设置下均优于Gradient-TD，尤其在λ值较小时表现更优。
• 使用重要性采样对整个TD误差项进行校正始终优于或至少匹配仅对v(S_t)进行部分校正，因此推荐采用完整校正。
• ETD(λ)对λ超参数的敏感性显著低于其他方法，在高方差环境下也表现良好。
• V-trace(λ)和Tree Backup(λ)互不占优；性能取决于环境特性，V-trace在高方差场景中通常表现更优。
• ABTD(ζ)表现与最佳方法相当，且在高方差任务中有时优于V-trace和Tree Backup。
• ETD(λ,β)并未明显优于ETD(λ)，其性能在β ≈ γ时最佳，此时退化为ETD(λ)。