[论文解读] Ecological Reinforcement Learning

John D. Co-Reyes, Suvansh Sanjeev+3 more|arXiv (Cornell University)|Jun 22, 2020

Reinforcement Learning in Robotics参考文献 46被引用 12

一句话总结

本文提出生态强化学习，主张环境特性——特别是环境塑造与动态性——可显著简化非周期性、稀疏奖励的强化学习。通过修改初始状态或增加转移熵，智能体在无需重置或密集奖励的情况下实现更优探索与学习，且训练出的策略在原始、结构较松散的MDP上泛化能力更强。

ABSTRACT

Much of the current work on reinforcement learning studies episodic settings, where the agent is reset between trials to an initial state distribution, often with well-shaped reward functions. Non-episodic settings, where the agent must learn through continuous interaction with the world without resets, and where the agent receives only delayed and sparse reward signals, is substantially more difficult, but arguably more realistic considering real-world environments do not present the learner with a convenient "reset mechanism" and easy reward shaping. In this paper, instead of studying algorithmic improvements that can address such non-episodic and sparse reward settings, we instead study the kinds of environment properties that can make learning under such conditions easier. Understanding how properties of the environment impact the performance of reinforcement learning agents can help us to structure our tasks in ways that make learning tractable. We first discuss what we term "environment shaping" -- modifications to the environment that provide an alternative to reward shaping, and may be easier to implement. We then discuss an even simpler property that we refer to as "dynamism," which describes the degree to which the environment changes independent of the agent's actions and can be measured by environment transition entropy. Surprisingly, we find that even this property can substantially alleviate the challenges associated with non-episodic RL in sparse reward settings. We provide an empirical evaluation on a set of new tasks focused on non-episodic learning with sparse rewards. Through this study, we hope to shift the focus of the community towards analyzing how properties of the environment can affect learning and the ultimate type of behavior that is learned via RL.

研究动机与目标

• 研究环境特性如何影响非周期性、稀疏奖励设定下的强化学习性能。
• 解决在真实环境中奖励塑造不切实际时，持续性、无重置学习的挑战。
• 探讨环境修改是否可作为引导学习的可行替代方案，替代奖励塑造。
• 评估提高环境动态性（转移熵）是否能在无显式课程或塑造的情况下稳定学习。
• 证明在生态化塑造环境中训练的智能体，可更好地泛化至原始、结构较松散的MDP。

提出的方法

• 引入‘环境塑造’——通过修改初始状态或动力学特性，引导智能体朝向高奖励区域，而不改变奖励函数。
• 提出‘动态性’作为可度量的属性，定义为环境转移熵，反映与智能体行为无关的随机性。
• 在新型非周期性、稀疏奖励任务上，实证评估环境塑造与提升动态性的影响。
• 使用状态访问频次与轨迹可视化，比较不同方法在探索效率与行为模式上的差异。
• 在修改后的环境中训练智能体，并在原始、未修改的MDP中评估其性能，以衡量泛化能力。
• 将环境塑造与奖励塑造（如一次性奖励或距离奖励）进行对比，以隔离奖励设计引入的偏差。

实验结果

研究问题

• RQ1 能否通过环境塑造（修改初始状态或动力学）改善非周期性、稀疏奖励强化学习中的学习效果？
• RQ2 提高环境动态性（转移熵）是否能在无显式课程或奖励塑造的情况下稳定学习并提升性能？
• RQ3 在探索效率与行为质量方面，环境塑造与动态性相较于传统奖励塑造有何差异？
• RQ4 在生态化塑造环境中训练的智能体，是否能更好地泛化至原始、结构较松散的MDP？
• RQ5 不同塑造方法会产生何种行为偏差？这些偏差如何影响非周期性设定下的任务完成？

主要发现

• 环境塑造可缩短随机策略到达高奖励区域的命中时间，在非周期性设定中提升探索效率与状态覆盖度。
• 提高环境动态性（更高转移熵）可稳定无重置学习，即使缺乏显式课程，也能实现有效训练。
• 通过环境塑造或动态性训练的智能体，在网格中实现更均匀的状态访问，避免陷入角落的困境。
• 如一次性奖励或距离奖励等奖励塑造方法会引入次优行为——例如重复动作或任务失败——这是由奖励函数偏差所致。
• 在形状化或动态环境中训练的策略，泛化至原始MDP时表现更优，能更快、更可靠地完成任务，优于标准奖励塑造训练的策略。
• 在狩猎环境中，环境塑造的智能体在15个时间步内成功完成任务，而奖励塑造的智能体在100个时间步内即因行为偏差而失败。