[Paper Review] Pixel-Attentive Policy Gradient for Multi-Fingered Grasping in Cluttered Scenes

Bohan Wu, Iretiayo Akinola+1 more|arXiv (Cornell University)|Mar 8, 2019

Robot Manipulation and Learning29 references4 citations

TL;DR

This paper proposes a pixel-attentive policy gradient method for end-to-end reinforcement learning of full 6-DOF multi-fingered grasps from a single depth image in cluttered scenes. By operating directly in pixel space and using a learnable attention mechanism to iteratively focus on promising regions, the method achieves 91.9% success rate on novel cluttered objects in the real world with no real-world fine-tuning, demonstrating strong sim-to-real transfer and robustness to occlusion.

ABSTRACT

Recent advances in on-policy reinforcement learning (RL) methods enabled learning agents in virtual environments to master complex tasks with high-dimensional and continuous observation and action spaces. However, leveraging this family of algorithms in multi-fingered robotic grasping remains a challenge due to large sim-to-real fidelity gaps and the high sample complexity of on-policy RL algorithms. This work aims to bridge these gaps by first reinforcement-learning a multi-fingered robotic grasping policy in simulation that operates in the pixel space of the input: a single depth image. Using a mapping from pixel space to Cartesian space according to the depth map, this method transfers to the real world with high fidelity and introduces a novel attention mechanism that substantially improves grasp success rate in cluttered environments. Finally, the direct-generative nature of this method allows learning of multi-fingered grasps that have flexible end-effector positions, orientations and rotations, as well as all degrees of freedom of the hand.

Motivation & Objective

• To address the high sample complexity and sim-to-real gap in on-policy reinforcement learning for multi-fingered robotic grasping.
• To enable full 6-DOF grasp policy learning (position, orientation, and all finger joint angles) directly from a single depth image.
• To improve grasp success in cluttered environments by introducing a learnable attention mechanism that progressively zooms into relevant image regions.
• To achieve robust real-world performance without real-world fine-tuning by minimizing sim-to-real domain shift through depth-only observation.

Proposed method

• The method uses a policy gradient reinforcement learning algorithm trained entirely in simulation using a single depth image as input.
• Grasp pose and finger joint angles are predicted per-pixel in image space and converted to 3D Cartesian coordinates using depth map-based point cloud reconstruction.
• A novel pixel-attentive mechanism enables iterative cropping of the depth image to focus on local regions, simulating zooming-in to improve grasp quality in clutter.
• The attention mechanism learns to stop cropping when a high-quality grasp is predicted, enabling non-top-down grasps with flexible end-effector poses.
• Domain randomization is applied during training by varying the number and arrangement of objects in the scene to improve robustness to clutter.
• The policy is trained end-to-end without any human-annotated grasp data or pre-defined grasp candidates.

Figure 1 : Pixel-Attentive Policy Gradient Multi-Fingered Grasping. Given a scene of cluttered objects (a) , our method takes in a single depth image and gradually zooms into a local region of the image to generate a good grasp. (b) , (c) and (d) show the zooming process, in which the green bounding

Experimental results

Research questions

• RQ1 Can a reinforcement learning policy trained solely on depth images achieve high real-world grasp success rates in cluttered scenes without real-world fine-tuning?
• RQ2 How does a learnable attention mechanism that progressively focuses on image regions improve grasp performance in cluttered environments compared to global observation?
• RQ3 To what extent does learning full 6-DOF grasp poses and multi-fingered configurations outperform top-down or two-fingered grasping in complex scenes?
• RQ4 How robust is the method to variations in camera viewpoint, such as non-top-down angles?
• RQ5 What is the contribution of multi-fingered grasping versus simplified grasp types (e.g., parallel or top-down) to overall performance?

Key findings

• The method achieved 96.7% success rate on single seen objects and 93.3% on single novel objects in the real world, demonstrating strong generalization.
• In cluttered scenes, the method achieved 92.9% success on seen objects and 91.9% on novel objects, indicating high robustness to occlusion and clutter.
• Ablation studies showed that removing the attention mechanism caused a 21.6% drop in success rate on cluttered seen objects, highlighting its critical role in handling occlusion.
• Enforcing top-down grasps reduced performance by 17.7% on cluttered seen objects, confirming the advantage of non-top-down grasps in complex scenes.
• Using only two-fingered grasps instead of multi-fingered hands caused a 43.4% performance drop on single-seen objects, demonstrating the significant benefit of full multi-fingered control.
• The method showed robustness to non-top-down camera angles, with 60° viewing angle performance statistically indistinguishable from top-down in simulation.

(a) PyBullet (Train)