[Paper Review] OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics

Peiqi Liu, Yaswanth Orru+4 more|arXiv (Cornell University)|Jan 22, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

OK-Robot proposes a zero-shot, open-knowledge robotics framework that integrates vision-language models (VLMs) with pre-trained navigation and grasping primitives for open-vocabulary pick-and-drop tasks in unstructured home environments. Without any training, it achieves a 58.5% success rate in real-world deployment across 10 homes, rising to 82% in cleaner settings, demonstrating that component integration and system-level heuristics are critical for performance.

ABSTRACT

Remarkable progress has been made in recent years in the fields of vision, language, and robotics. We now have vision models capable of recognizing objects based on language queries, navigation systems that can effectively control mobile systems, and grasping models that can handle a wide range of objects. Despite these advancements, general-purpose applications of robotics still lag behind, even though they rely on these fundamental capabilities of recognition, navigation, and grasping. In this paper, we adopt a systems-first approach to develop a new Open Knowledge-based robotics framework called OK-Robot. By combining Vision-Language Models (VLMs) for object detection, navigation primitives for movement, and grasping primitives for object manipulation, OK-Robot offers a integrated solution for pick-and-drop operations without requiring any training. To evaluate its performance, we run OK-Robot in 10 real-world home environments. The results demonstrate that OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks, representing a new state-of-the-art in Open Vocabulary Mobile Manipulation (OVMM) with nearly 1.8x the performance of prior work. On cleaner, uncluttered environments, OK-Robot's performance increases to 82%. However, the most important insight gained from OK-Robot is the critical role of nuanced details when combining Open Knowledge systems like VLMs with robotic modules. Videos of our experiments and code are available on our website: https://ok-robot.github.io

Motivation & Objective

• To develop a general-purpose robotics framework that enables zero-shot open-vocabulary mobile manipulation in unstructured home environments.
• To investigate how pre-trained open-knowledge models (VLMs, navigation, grasping) can be effectively combined without task-specific fine-tuning.
• To identify and address system-level failure modes arising from component-level inaccuracies in vision, navigation, and grasping modules.
• To establish a benchmark for open-vocabulary mobile manipulation (OVMM) using real-world deployment in diverse, unseen homes.
• To support future research by open-sourcing code and modules for reproducibility and extensibility.

Proposed method

• OK-Robot uses a semantic memory built from dense vision-language representations via LangSam and CLIP on iPhone-sourced scans of new homes.
• It matches natural language queries to objects in the semantic memory using language embeddings, enabling open-vocabulary object localization.
• Navigation is performed using an A*-based path planner (similar to USANet) to reach target locations derived from VLM outputs.
• Grasping is executed using a pre-trained grasping policy that operates directly on detected object locations without retraining.
• A state-machine controller sequences object localization, navigation, and grasping primitives in a modular, zero-shot pipeline.
• Heuristics are applied to handle robot-specific limitations, such as avoiding collisions and adjusting grasp poses based on physical constraints.

Figure 2: Open-vocabulary, open knowledge object localization and navigation in the real-world. We use the VoxelMap [ 25 ] for localizing objects with natural language queries, and use an A* algorithm similar to USANet [ 26 ] for path planning.

Experimental results

Research questions

• RQ1 How effective is a zero-shot integration of pre-trained VLMs, navigation, and grasping models for open-vocabulary mobile manipulation in real-world homes?
• RQ2 What role do system-level design choices—such as component composition and heuristics—play in overall success, independent of individual model accuracy?
• RQ3 How do environmental factors like clutter, object transparency, or size affect the performance of open-knowledge robotics systems?
• RQ4 To what extent can failure modes be mitigated through improved error detection, retry mechanisms, or user interaction?
• RQ5 What are the key bottlenecks in current open-knowledge robotics pipelines, and how can they be addressed through future model or hardware improvements?

Key findings

• OK-Robot achieves a 58.5% success rate in open-ended pick-and-drop tasks across 10 real-world, previously unseen home environments, setting a new state-of-the-art in open-vocabulary mobile manipulation (OVMM).
• In cleaner, uncluttered environments, the success rate increases to 82.4%, highlighting the strong impact of environmental quality on system performance.
• Pre-trained VLMs such as CLIP and OWL-ViT enable effective zero-shot object localization and navigation when combined with semantic memory and language matching.
• Pre-trained grasping models can be directly applied without fine-tuning, showing that such models generalize well to unseen home scenarios.
• The integration strategy—particularly the use of heuristics to handle robot kinematics and physical constraints—significantly improves success rates over naive component stacking.
• Failure accumulation across modules remains a major challenge, as even 80% accurate components can result in less than 60% overall success due to multiplicative error effects.

Figure 3: Open-vocabulary grasping in the real world. From left to right, we show the (a) robot POV image, (b) all suggested grasps from AnyGrasp [ 19 ] , (c) object mask given label from LangSam [ 24 ] , (d) grasp points filtered by the mask, and (e) grasp chosen for execution.