[Paper Review] Semantic Mechanical Search with Large Vision and Language Models

Satvik Sharma, Huang Huang+5 more|arXiv (Cornell University)|Feb 24, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Semantic Mechanical Search (SMS), a framework that uses large language models (LLMs) to generate semantic occupancy distributions for improving mechanical search in robotics. By decoupling semantic reasoning from geometric planning, SMS enhances performance by 24% in simulation and 47.1% in physical experiments across pharmacy, kitchen, and office environments, outperforming CLIP-based baselines through deeper semantic reasoning over object relationships.

ABSTRACT

Moving objects to find a fully-occluded target object, known as mechanical search, is a challenging problem in robotics. As objects are often organized semantically, we conjecture that semantic information about object relationships can facilitate mechanical search and reduce search time. Large pretrained vision and language models (VLMs and LLMs) have shown promise in generalizing to uncommon objects and previously unseen real-world environments. In this work, we propose a novel framework called Semantic Mechanical Search (SMS). SMS conducts scene understanding and generates a semantic occupancy distribution explicitly using LLMs. Compared to methods that rely on visual similarities offered by CLIP embeddings, SMS leverages the deep reasoning capabilities of LLMs. Unlike prior work that uses VLMs and LLMs as end-to-end planners, which may not integrate well with specialized geometric planners, SMS can serve as a plug-in semantic module for downstream manipulation or navigation policies. For mechanical search in closed-world settings such as shelves, we compare with a geometric-based planner and show that SMS improves mechanical search performance by 24% across the pharmacy, kitchen, and office domains in simulation and 47.1% in physical experiments. For open-world real environments, SMS can produce better semantic distributions compared to CLIP-based methods, with the potential to be integrated with downstream navigation policies to improve object navigation tasks. Code, data, videos, and the appendix are available: https://sites.google.com/view/semantic-mechanical-search

Motivation & Objective

• To improve mechanical search in robotics by leveraging semantic relationships between objects in occluded environments.
• To investigate whether LLMs can generate more effective semantic representations than vision-language models like CLIP for object search tasks.
• To develop a plug-in semantic module that integrates with existing geometric planners for mechanical search.
• To evaluate SMS in both closed-world (known object lists) and open-world (unseen objects) environments.
• To demonstrate that decoupling semantic reasoning from visual perception improves search performance.

Proposed method

• SMS uses object detection or instance segmentation to identify objects in a scene, then applies image captioning when object lists are unavailable.
• It employs a vision-language model (VLM) to encode visual features into text embeddings, which are then processed by a large language model (LLM) to infer semantic affinities between detected objects and the target.
• The LLM generates a semantic occupancy distribution by reasoning over object relationships, such as co-location patterns (e.g., toothpaste near toothbrushes).
• This distribution is used as input to downstream geometric planners, enabling semantic-aware mechanical search without modifying the core planning algorithm.
• For closed-world settings, object lists are used to guide detection and captioning; for open-world, segmentation and captioning are used to describe object masks.
• The framework is evaluated using both simulation and real-world experiments on shelves and mobile navigation tasks.

Experimental results

Research questions

• RQ1 Can semantic reasoning via LLMs improve mechanical search performance in occluded environments?
• RQ2 Does generating a semantic occupancy distribution using LLMs outperform CLIP-based visual-semantic matching in mechanical search?
• RQ3 Can a decoupled semantic module be effectively integrated with existing geometric planners for mechanical search?
• RQ4 How does SMS perform in open-world settings where object lists are unavailable?
• RQ5 What is the impact of using LLMs to reason over language-embedded visual features compared to end-to-end VLM-based planning?

Key findings

• SMS improved mechanical search performance by 24% across pharmacy, kitchen, and office domains in simulation, compared to a geometric-only baseline.
• In physical experiments, SMS achieved a 47.1% improvement in success rate over the same baseline, demonstrating strong real-world generalization.
• SMS outperformed CLIP-based methods in open-world navigation tasks, correctly identifying the target stack of books while CLIP failed due to bag-of-words limitations.
• The semantic distribution generated by SMS was more robust to distractors—e.g., CLIP incorrectly associated 'stack' with pillars, while SMS correctly identified the desk area.
• Decoupling visual perception from semantic reasoning via LLMs led to better performance than end-to-end VLM-based planning, as LLMs better capture relational semantics.
• The framework successfully integrated with downstream geometric planners, showing compatibility and modularity in real-world robotic systems.