[论文解读] Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning

Mohammad Babaeizadeh, Mohammad Saffar+5 more|arXiv (Cornell University)|Dec 8, 2020

Reinforcement Learning in Robotics参考文献 50被引用 4

一句话总结

本文通过比较仅预测奖励的模型与预测未来观测（图像）的模型，评估了视觉模型基于强化学习中的设计权衡。研究发现，预测图像能显著提升任务性能，且图像预测准确率与下游性能的相关性高于奖励预测准确率，凸显了视觉建模在探索与规划效率中的关键作用。

ABSTRACT

Model-based reinforcement learning (MBRL) methods have shown strong sample efficiency and performance across a variety of tasks, including when faced with high-dimensional visual observations. These methods learn to predict the environment dynamics and expected reward from interaction and use this predictive model to plan and perform the task. However, MBRL methods vary in their fundamental design choices, and there is no strong consensus in the literature on how these design decisions affect performance. In this paper, we study a number of design decisions for the predictive model in visual MBRL algorithms, focusing specifically on methods that use a predictive model for planning. We find that a range of design decisions that are often considered crucial, such as the use of latent spaces, have little effect on task performance. A big exception to this finding is that predicting future observations (i.e., images) leads to significant task performance improvement compared to only predicting rewards. We also empirically find that image prediction accuracy, somewhat surprisingly, correlates more strongly with downstream task performance than reward prediction accuracy. We show how this phenomenon is related to exploration and how some of the lower-scoring models on standard benchmarks (that require exploration) will perform the same as the best-performing models when trained on the same training data. Simultaneously, in the absence of exploration, models that fit the data better usually perform better on the downstream task as well, but surprisingly, these are often not the same models that perform the best when learning and exploring from scratch. These findings suggest that performance and exploration place important and potentially contradictory requirements on the model.

研究动机与目标

• 理解视觉模型基于强化学习（MBRL）中的基本设计选择如何影响性能。
• 评估预测未来观测（图像）与仅预测奖励对下游任务性能的影响。
• 探究潜在空间建模或其他架构选择是否显著影响性能。
• 分析模型准确率、规划与探索在MBRL智能体中的相互作用。
• 提供独立于算法混淆因素的、清晰且受控的模型设计比较。

提出的方法

• 作者比较了多种在是否预测图像或仅预测奖励，以及动力学是否在潜在空间或观测空间中建模方面不同的MBRL模型架构。
• 所有模型均使用相同的规划算法（CEM）和固定超参数进行训练与评估，以隔离模型设计的影响。
• 研究在多个环境中采用一致的训练与评估协议，重点关注控制性能和预测准确率指标。
• 模型在标准基准（需要探索）和固定训练数据上进行评估，以将性能与探索动态解耦。
• 测量并关联图像与奖励预测准确率与下游任务性能，以评估预测保真度。
• 分析包括对模型容量的消融研究，如缩小架构，以评估建模容量对性能的影响。

实验结果

研究问题

• RQ1 在视觉MBRL中，预测未来观测（图像）是否比仅预测奖励带来更好的任务性能？
• RQ2 图像预测准确率与奖励预测准确率相比，在预测下游任务性能方面表现如何？
• RQ3 使用潜在空间或自回归建模等架构选择在多大程度上影响性能？
• RQ4 性能与探索如何对预测模型施加相互冲突的要求？
• RQ5 当从零开始学习时，是否拟合数据更好的模型总是表现最佳？探索是否会改变这一关系？

主要发现

• 即使后者已提供足够的控制信息，预测未来观测（图像）仍能显著提升任务性能，优于仅预测奖励。
• 图像预测准确率与下游任务性能的相关性高于奖励预测准确率，表明视觉保真度是规划质量的关键代理指标。
• 在标准基准（需要探索）上表现较差的模型，当在相同固定数据上训练时，常能与顶尖模型性能相当，表明其对探索的依赖性。
• 在缺乏探索的情况下，具有更高数据拟合能力（即更高预测准确率）的模型通常表现更好，但这些模型在从零开始学习时并不总是表现最佳。
• 使用潜在空间或特定网络架构等设计选择对最终任务性能的影响微乎其微，挑战了其必要性的假设。
• 研究结果表明，性能与探索对预测模型施加了潜在矛盾的要求，而图像预测能更好地支持探索与长期规划。