[Paper Review] Efficient learning of goal-oriented push-grasping synergy in clutter

Kechun Xu, Hongxiang Yu+3 more|arXiv (Cornell University)|Mar 9, 2021

Robot Manipulation and Learning36 references4 citations

TL;DR

This paper proposes a goal-conditioned hierarchical reinforcement learning framework that improves sample efficiency in goal-oriented push-grasping tasks within cluttered environments. By using goal relabeling and adversarial training—where grasping acts as a discriminator to guide pushing—the method densifies rewards and enables faster, more effective learning. The approach achieves higher task completion and grasp success rates with fewer actions, generalizes to real-world settings without fine-tuning, and performs well under both goal-oriented and goal-agnostic conditions.

ABSTRACT

We focus on the task of goal-oriented grasping, in which a robot is supposed to grasp a pre-assigned goal object in clutter and needs some pre-grasp actions such as pushes to enable stable grasps. However, in this task, the robot gets positive rewards from environment only when successfully grasping the goal object. Besides, joint pushing and grasping elongates the action sequence, compounding the problem of reward delay. Thus, sample inefficiency remains a main challenge in this task. In this paper, a goal-conditioned hierarchical reinforcement learning formulation with high sample efficiency is proposed to learn a push-grasping policy for grasping a specific object in clutter. In our work, sample efficiency is improved by two means. First, we use a goal-conditioned mechanism by goal relabeling to enrich the replay buffer. Second, the pushing and grasping policies are respectively regarded as a generator and a discriminator and the pushing policy is trained with supervision of the grasping discriminator, thus densifying pushing rewards. To deal with the problem of distribution mismatch caused by different training settings of two policies, an alternating training stage is added to learn pushing and grasping in turn. A series of experiments carried out in simulation and real world indicate that our method can quickly learn effective pushing and grasping policies and outperforms existing methods in task completion rate and goal grasp success rate by less times of motion. Furthermore, we validate that our system can also adapt to goal-agnostic conditions with better performance. Note that our system can be transferred to the real world without any fine-tuning. Our code is available at https://github.com/xukechun/Efficient_goal-oriented_push-grasping_synergy.

Motivation & Objective

• To address the challenge of sparse rewards and sample inefficiency in goal-oriented push-grasping tasks within cluttered scenes.
• To enable efficient learning of synergistic pushing and grasping policies that can isolate and grasp a specific target object.
• To improve generalization from simulation to real-world deployment without fine-tuning.
• To extend applicability to goal-agnostic scenarios by removing target-specific conditioning.

Proposed method

• Formulates the task as a goal-conditioned hierarchical reinforcement learning problem to structure pushing and grasping as distinct but coordinated policies.
• Employs goal relabeling in the replay buffer to enrich experience data, improving sample efficiency by reassigning goals to successfully grasped non-target objects.
• Introduces an adversarial training mechanism where the grasping policy acts as a discriminator to provide dense, informative rewards for pushing actions.
• Uses alternating training between pushing and grasping policies to mitigate distribution shift caused by differing training dynamics.
• Trains the system end-to-end in simulation and transfers it directly to real-world environments without additional fine-tuning.
• Applies a goal mask to the observation space to condition the policy on the target object, with the option to remove this for goal-agnostic operation.

Experimental results

Research questions

• RQ1 How can sample efficiency be improved in goal-oriented push-grasping tasks with sparse dense rewards?
• RQ2 Can adversarial training between pushing and grasping policies densify rewards and improve policy learning?
• RQ3 How does alternating training between pushing and grasping policies mitigate distribution shift in hierarchical RL?
• RQ4 To what extent can a policy trained in simulation generalize to real-world environments without fine-tuning?
• RQ5 Can the same framework perform effectively in both goal-oriented and goal-agnostic grasping scenarios?

Key findings

• In simulation, the proposed method achieved a 93.3% task completion rate and 86.6% grasp success rate on challenging arrangements, outperforming baselines like Grasping the Invisible.
• The method reduced average motion count to 4.62 actions per task in challenging cases, showing improved action efficiency.
• In real-world experiments, the system achieved 93.3% completion and 86.6% grasp success rate on challenging arrangements, surpassing the baseline method (86.7% and 75.2%, respectively).
• The system generalized to novel, unseen objects with complex shapes, though failure cases occurred with completely unfamiliar shapes.
• The method achieved a 12.4% higher completion rate and 5.3% higher grasp success rate than VPG in goal-agnostic conditions.
• The system successfully transferred from simulation to real world without any fine-tuning, demonstrating strong zero-shot generalization.