[论文解读] Lan-grasp: Using Large Language Models for Semantic Object Grasping and Placement

Reihaneh Mirjalili, Michael Krawez+4 more|arXiv (Cornell University)|Oct 8, 2023

Multimodal Machine Learning Applications被引用 5

一句话总结

LAN-grasp 提出了一种零样本语义抓取方法，利用大型语言模型（LLMs）和视觉-语言模型（VLMs）在不进行微调的情况下识别日常物体上的最优抓取区域。通过提示 GPT-4 选择安全且功能合适的抓取部位，并利用 OWL-ViT 进行视觉定位，该系统生成了符合人类偏好的抓取方式，其平均相似度达到 0.94，显著优于传统方法和语义基线方法。

ABSTRACT

In this paper, we propose Lan-grasp, a novel approach towards more appropriate semantic grasping and placing. We leverage foundation models to equip the robot with a semantic understanding of object geometry, enabling it to identify the right place to grasp, which parts to avoid, and the natural pose for placement. This is an important contribution to grasping and utilizing objects in a more meaningful and safe manner. We leverage a combination of a Large Language Model, a Vision-Language Model, and a traditional grasp planner to generate grasps that demonstrate a deeper semantic understanding of the objects. Building on foundation models provides us with a zero-shot grasp method that can handle a wide range of objects without requiring further training or fine-tuning. We also propose a method for safely putting down a grasped object. The core idea is to rotate the object upright utilizing a pretrained generative model and the reasoning capabilities of a VLM. We evaluate our method in real-world experiments on a custom object dataset and present the results of a survey that asks participants to choose an object part appropriate for grasping. The results show that the grasps generated by our method are consistently ranked higher by the participants than those generated by a conventional grasping planner and a recent semantic grasping approach. In addition, we propose a Visual Chain-of-Thought feedback loop to assess grasp feasibility in complex scenarios. This mechanism enables dynamic reasoning and generates alternative grasp strategies when needed, ensuring safer and more effective grasping outcomes.

研究动机与目标

• 使机器人能够以尊重物体语义功能和设计的方式抓取日常物品，确保安全性和可用性。
• 通过利用基础模型实现零样本推理，消除对特定任务训练或微调的需求，以识别可抓取部位。
• 通过整合物体部件的语义知识和预期用途，改进仅依赖几何信息的抓取方法。
• 评估基于基础模型的抓取选择是否与真实场景中人类的直觉偏好一致。

提出的方法

• 该方法使用大型语言模型（GPT-4）基于语义理解与预期用途，推理物体中适合抓取的部位。
• 使用视觉-语言模型（OWL-ViT）将 LLM 识别出的部位在物体的 RGB 图像中进行定位。
• 仅在 VLM 识别出的区域内生成抓取建议，确保抓取的语义合理性。
• 该方法模块化且兼容任意 LLM 和 VLM，可轻松适配不同基础模型。
• 使用现成的抓取规划器在语义区域内生成稳定抓取，确保物理可行性。
• 该方法以零样本方式运行，无需在新物体类别上进行额外训练或微调。

Figure 1: Robot performing the command of “Pick up the ice cream please” . The grasp on the left is generated without including semantic information while the grasp on the right is performed using our method leveraging a deeper understanding of the task and the object provided by Large Language Mode

实验结果

研究问题

• RQ1 能否在不进行微调的情况下，利用 LLM 和 VLM 等基础模型识别日常物体上语义上合适的抓取区域？
• RQ2 LLM 生成的抓取区域与人类在物体操作中的直觉偏好有多吻合？
• RQ3 视觉-语言模型能否在物体的真实图像中准确定位 LLM 提出的语义抓取建议？
• RQ4 所提出的方法在人类偏好评估中是否优于仅依赖几何信息和现有语义抓取基线方法？
• RQ5 该系统能否泛化到推理过程中未见过的物体类别？

主要发现

• LAN-grasp 在所有测试物体上实现了与人类抓取偏好 0.94 的平均相似度，显著优于 GraspIt!（0.31）和 GraspGPT（0.67）。
• 在所有测试案例中，LAN-grasp 选择的抓取区域均与多数人类偏好一致，显示出与人类直觉行为的高度一致。
• 该方法成功识别了多样化物体的合适抓取区域，包括工具、水杯以及冰淇淋和植物等易碎物品。
• LLM 正确建议避开遥控器上的按钮和智能手机屏幕等部位，显示出对安全性的意识。
• 初步结果显示，LLM 还能识别应避免的部位（如电风扇的扇叶、茶壶的壶嘴），表明其在安全感知抓取方面的潜力。
• 该方法的零样本特性使其无需重新训练或微调，即可在广泛未见过的家庭物品上实现良好泛化。

Figure 2: Our approach in a nutshell: The command from the user is turned into a prompt suitable for the Large Language Model (LLM). With this prompt as an input, the large language model outputs the proper part for grasping the object, which in this example is the cone . This word is then grounded