[论文解读] Unsupervised Perceptual Rewards for Imitation Learning

Pierre Sermanet, Kelvin Xu+1 more|arXiv (Cornell University)|Dec 20, 2016

Reinforcement Learning in Robotics参考文献 25被引用 5

一句话总结

本文提出一种无监督方法，仅通过少量人类示范视频，利用预训练的深度视觉特征来学习密集的、感知性的奖励函数。该方法无需监督即可自动发现中间子目标和判别性视觉特征，使真实机器人仅通过人类姿态视频示范即可成功学习复杂的操作技能（如开门），其性能与使用真实奖励信号时相当。

ABSTRACT

Reward function design and exploration time are arguably the biggest obstacles to the deployment of reinforcement learning (RL) agents in the real world. In many real-world tasks, designing a reward function takes considerable hand engineering and often requires additional sensors to be installed just to measure whether the task has been executed successfully. Furthermore, many interesting tasks consist of multiple implicit intermediate steps that must be executed in sequence. Even when the final outcome can be measured, it does not necessarily provide feedback on these intermediate steps. To address these issues, we propose leveraging the abstraction power of intermediate visual representations learned by deep models to quickly infer perceptual reward functions from small numbers of demonstrations. We present a method that is able to identify key intermediate steps of a task from only a handful of demonstration sequences, and automatically identify the most discriminative features for identifying these steps. This method makes use of the features in a pre-trained deep model, but does not require any explicit specification of sub-goals. The resulting reward functions can then be used by an RL agent to learn to perform the task in real-world settings. To evaluate the learned reward, we present qualitative results on two real-world tasks and a quantitative evaluation against a human-designed reward function. We also show that our method can be used to learn a real-world door opening skill using a real robot, even when the demonstration used for reward learning is provided by a human using their own hand. To our knowledge, these are the first results showing that complex robotic manipulation skills can be learned directly and without supervised labels from a video of a human performing the task. Supplementary material and data are available at https://sermanet.github.io/rewards

研究动机与目标

• 解决真实世界强化学习中稀疏且人工设计的奖励函数问题。
• 实现在无需显式子目标标注的情况下，仅通过极少人类示范进行模仿学习。
• 从视频示范中自动发现中间子目标和判别性视觉特征。
• 仅使用从人类示范中衍生的感知奖励，实现真实世界机器人策略学习。
• 证明预训练视觉特征在未见场景和机器人本体上具有良好的泛化能力。

提出的方法

• 使用预训练的深度神经网络（例如 Inception）从示范视频中提取中层和高层视觉特征。
• 应用聚类或分割方法，识别示范序列中的关键中间步骤。
• 在视觉特征之上训练分类器，以区分每个中间步骤，最大化所有视频之间的判别性。
• 将各步骤特定的分类器组合成单一、平滑的密集奖励函数。
• 在强化学习算法（例如 PI2）中使用所得的感知奖励函数来训练真实机器人。
• 无需动力学示范、状态传感器或显式的子目标标注。

实验结果

研究问题

• RQ1 能否仅从少量人类示范视频中直接学习感知奖励函数，而无需任何子目标标注？
• RQ2 预训练视觉特征是否足够泛化，以在不重新训练的情况下表示新场景中的子目标和目标？
• RQ3 所发现的中间步骤和奖励函数能否使真实机器人学会复杂操作任务（如开门）？
• RQ4 无监督感知奖励的性能与通过传感器（如IMU）获得的真实奖励信号相比如何？
• RQ5 该方法能否在物体外观和示范风格（如人类与机器人、不同门）变化的情况下实现泛化？

主要发现

• 该方法仅从人类示范视频中成功学习到了感知奖励函数，使真实机器人在无需真实奖励或动力学数据的情况下学会了开门。
• 机器人的成功率与使用IMU传感器获取真实奖励信号的PI2基线方法相当。
• 奖励函数在训练期间未见过的不同门外观（白色、黑色、蓝色）和方向上均表现出良好泛化能力。
• 即使机器人本体与示范者不同，该方法仍能从人类示范中发现4至5个中间子目标。
• 所学习的奖励函数有效支持策略学习，训练初期即能快速找到高奖励轨迹，表明信号质量优异。
• 该方法对视觉变化具有鲁棒性，并能仅用少量示范即泛化到此前未见过的场景。