[Paper Review] ThinkGrasp: A Vision-Language System for Strategic Part Grasping in Clutter

Yaoyao Qian, Xupeng Zhu+6 more|arXiv (Cornell University)|Jul 16, 2024

Natural Language Processing Techniques4 citations

TL;DR

ThinkGrasp is a vision-language robotic grasping system that uses GPT-4o’s reasoning to plan strategic, multi-step grasping in heavily cluttered environments. By combining language-guided object prioritization, occlusion handling, and segmentation via LangSAM and VLPart, it achieves 98.0% success in simulation and 75–90% success in real-world tasks with few steps, outperforming prior methods in generalization and robustness.

ABSTRACT

Robotic grasping in cluttered environments remains a significant challenge due to occlusions and complex object arrangements. We have developed ThinkGrasp, a plug-and-play vision-language grasping system that makes use of GPT-4o's advanced contextual reasoning for heavy clutter environment grasping strategies. ThinkGrasp can effectively identify and generate grasp poses for target objects, even when they are heavily obstructed or nearly invisible, by using goal-oriented language to guide the removal of obstructing objects. This approach progressively uncovers the target object and ultimately grasps it with a few steps and a high success rate. In both simulated and real experiments, ThinkGrasp achieved a high success rate and significantly outperformed state-of-the-art methods in heavily cluttered environments or with diverse unseen objects, demonstrating strong generalization capabilities.

Motivation & Objective

• To address the challenge of robotic grasping in heavily cluttered environments where target objects are occluded or nearly invisible.
• To improve generalization and robustness in grasping unseen objects under heavy occlusion using language-conditioned reasoning.
• To develop a plug-and-play system that integrates vision-language models with segmentation and grasp planning for safe, efficient, and adaptive grasping.
• To reduce reliance on large-scale annotated datasets by leveraging the reasoning capabilities of large language models like GPT-4o.
• To enable real-time adaptation to new language goals and novel objects through prompt-based interaction.

Proposed method

• The system uses GPT-4o to interpret natural language instructions and generate a chain-of-thought reasoning plan for removing obstructing objects in a goal-oriented manner.
• It employs a 3×3 grid-based grasp point selection strategy on segmented object regions to improve grasp pose accuracy and safety.
• LangSAM and VLPart are used for robust image and point cloud segmentation, decoupling segmentation from language model predictions to prevent error propagation.
• The system uses a modular pipeline: language model for strategy, segmentation models for object localization, and a downstream grasp model (FGC-GraspNet) for pose generation.
• It integrates with MoveIt and ROS for motion planning and robotic control, using RRT* for path planning and a 6-DoF UR5 arm with Robotiq 85 gripper in real-world deployment.
• The system is deployed via Flask on a server with dual 3090 GPUs, with inference latency under 10 seconds using the GPT-4o API.

Experimental results

Research questions

• RQ1 Can a vision-language model like GPT-4o enable strategic, multi-step grasping in highly cluttered scenes with minimal human intervention?
• RQ2 How does integrating language reasoning with segmentation and grasp planning improve success rates in occluded environments?
• RQ3 To what extent can the system generalize to unseen objects and novel language instructions in both simulation and real-world settings?
• RQ4 What is the contribution of each component (e.g., GPT-4o, LangSAM, VLPart) to the overall system performance?
• RQ5 Can a plug-and-play system achieve high success rates with few steps across diverse, complex cluttered scenes?

Key findings

• In simulation, ThinkGrasp achieved a 98.0% success rate on the RefCOCO dataset under heavy clutter, significantly outperforming OVGNet (43.8%) and VLG (75.3%).
• Even with nearly invisible target objects, the system maintained a 78.9% success rate, demonstrating strong generalization to unseen objects and complex occlusions.
• In real-world experiments, the system achieved a 75% success rate in Step 1 and 80% in Step 2 for grasping tasks like 'get a tape' and 'get a knife by handle', outperforming VL-Grasp.
• The ablation study showed that removing GPT-4o or the 3×3 grasp grid reduced performance, confirming the importance of each component in the system's success.
• The system required fewer steps than baseline methods and maintained high reliability despite challenges from single-view reconstruction and robotic control variability.
• Failures were primarily due to low-quality grasp poses from the downstream model, image quality limitations, and robotic instability, highlighting the need for improved perception and control.