[论文解读] Efficient learning of goal-oriented push-grasping synergy in clutter

Kechun Xu, Hongxiang Yu+3 more|arXiv (Cornell University)|Mar 9, 2021

Robot Manipulation and Learning参考文献 36被引用 4

一句话总结

本文提出了一种目标条件化的分层强化学习框架，通过使用目标重标注和对抗性训练（其中抓取行为作为判别器来指导推送），在杂乱环境中的目标导向推送-抓取任务中提升了样本效率。该方法通过密集化奖励信号，实现了更快、更有效的学习。该方法在更少动作下实现了更高的任务完成率和抓取成功率，无需微调即可泛化至真实世界，并在目标导向和无目标导向条件下均表现出色。

ABSTRACT

We focus on the task of goal-oriented grasping, in which a robot is supposed to grasp a pre-assigned goal object in clutter and needs some pre-grasp actions such as pushes to enable stable grasps. However, in this task, the robot gets positive rewards from environment only when successfully grasping the goal object. Besides, joint pushing and grasping elongates the action sequence, compounding the problem of reward delay. Thus, sample inefficiency remains a main challenge in this task. In this paper, a goal-conditioned hierarchical reinforcement learning formulation with high sample efficiency is proposed to learn a push-grasping policy for grasping a specific object in clutter. In our work, sample efficiency is improved by two means. First, we use a goal-conditioned mechanism by goal relabeling to enrich the replay buffer. Second, the pushing and grasping policies are respectively regarded as a generator and a discriminator and the pushing policy is trained with supervision of the grasping discriminator, thus densifying pushing rewards. To deal with the problem of distribution mismatch caused by different training settings of two policies, an alternating training stage is added to learn pushing and grasping in turn. A series of experiments carried out in simulation and real world indicate that our method can quickly learn effective pushing and grasping policies and outperforms existing methods in task completion rate and goal grasp success rate by less times of motion. Furthermore, we validate that our system can also adapt to goal-agnostic conditions with better performance. Note that our system can be transferred to the real world without any fine-tuning. Our code is available at https://github.com/xukechun/Efficient_goal-oriented_push-grasping_synergy.

研究动机与目标

• 解决在杂乱场景中目标导向推送-抓取任务面临的稀疏奖励与样本效率低下问题。
• 实现协同推送与抓取策略的高效学习，能够隔离并抓取特定目标物体。
• 在无需微调的情况下，实现从仿真环境到真实世界部署的强泛化能力。
• 通过移除对目标特定条件的依赖，将适用性扩展至无目标导向场景。

提出的方法

• 将任务建模为一种目标条件化的分层强化学习问题，将推送与抓取作为独立但协调的策略进行结构化设计。
• 在经验回放缓冲区中采用目标重标注技术，丰富经验数据，通过将目标重新分配给成功抓取的非目标物体，提升样本效率。
• 引入对抗性训练机制，使抓取策略充当判别器，为推送动作提供密集且信息丰富的奖励信号。
• 通过交替训练推送与抓取策略，缓解因训练动态差异导致的分布偏移问题。
• 在仿真环境中端到端训练系统，并直接将其迁移到真实世界环境，无需额外微调。
• 在观测空间中应用目标掩码，以目标物体为条件进行策略控制，同时支持移除该条件以实现无目标导向运行。

实验结果

研究问题

• RQ1 在稀疏密集奖励的条件下，如何提升目标导向推送-抓取任务的样本效率？
• RQ2 推送与抓取策略之间的对抗性训练是否能够使奖励更密集，从而改善策略学习？
• RQ3 推送与抓取策略的交替训练在分层强化学习中如何缓解分布偏移问题？
• RQ4 在无需微调的情况下，仿真环境中训练的策略在真实世界中的泛化能力有多强？
• RQ5 该框架是否能在目标导向和无目标导向的抓取场景中均表现出色？

主要发现

• 在仿真环境中，所提方法在具有挑战性的布局下实现了93.3%的任务完成率和86.6%的抓取成功率，优于基线方法如Grasping the Invisible。
• 在复杂情况下，该方法将平均动作次数降低至每项任务4.62次，显示出更高的动作效率。
• 在真实世界实验中，系统在具有挑战性的布局下实现了93.3%的完成率和86.6%的抓取成功率，优于基线方法（分别为86.7%和75.2%）。
• 系统能够泛化至具有复杂形状的新颖、未见过的物体，尽管在完全不熟悉的形状上仍会出现失败案例。
• 在无目标导向条件下，该方法比VPG高出12.4%的完成率和5.3%的抓取成功率。
• 系统成功实现了从仿真到真实世界的零样本迁移，展现出强大的零样本泛化能力。