[Paper Review] INVIGORATE: Interactive Visual Grounding and Grasping in Clutter

Hanbo Zhang, Yunfan Lu+4 more|arXiv (Cornell University)|Aug 25, 2021

Multimodal Machine Learning Applications44 references4 citations

TL;DR

INVIGORATE is a robot system that integrates data-driven deep learning with model-based POMDP planning to enable interactive visual grounding and grasping in cluttered environments using natural language. By combining neural networks for object detection, visual grounding, question generation, and object blocking relationship estimation within an object-centric POMDP, it achieves 83% success rate in identifying and grasping target objects despite visual occlusions and linguistic ambiguity.

ABSTRACT

This paper presents INVIGORATE, a robot system that interacts with human through natural language and grasps a specified object in clutter. The objects may occlude, obstruct, or even stack on top of one another. INVIGORATE embodies several challenges: (i) infer the target object among other occluding objects, from input language expressions and RGB images, (ii) infer object blocking relationships (OBRs) from the images, and (iii) synthesize a multi-step plan to ask questions that disambiguate the target object and to grasp it successfully. We train separate neural networks for object detection, for visual grounding, for question generation, and for OBR detection and grasping. They allow for unrestricted object categories and language expressions, subject to the training datasets. However, errors in visual perception and ambiguity in human languages are inevitable and negatively impact the robot's performance. To overcome these uncertainties, we build a partially observable Markov decision process (POMDP) that integrates the learned neural network modules. Through approximate POMDP planning, the robot tracks the history of observations and asks disambiguation questions in order to achieve a near-optimal sequence of actions that identify and grasp the target object. INVIGORATE combines the benefits of model-based POMDP planning and data-driven deep learning. Preliminary experiments with INVIGORATE on a Fetch robot show significant benefits of this integrated approach to object grasping in clutter with natural language interactions. A demonstration video is available at https://youtu.be/zYakh80SGcU.

Motivation & Objective

• Address the challenge of robust object retrieval in cluttered environments where visual occlusions and linguistic ambiguity hinder performance.
• Integrate data-driven neural networks with model-based planning to handle perceptual uncertainty and language ambiguity in real-world robot interactions.
• Enable interactive disambiguation through question generation when visual grounding or language understanding is uncertain.
• Develop a system that reasons about object relationships and perceptual reliability to optimize action sequences under partial observability.
• Achieve high success rates in real-world robot grasping tasks despite noisy visual inputs and ambiguous language instructions.

Proposed method

• Train separate deep neural networks for object detection, visual grounding of referring expressions, object blocking relationship (OBR) detection, and disambiguation question generation.
• Formulate an object-centric partially observable Markov decision process (POMDP) that models noisy observations from neural networks and maintains a belief state over object identities and relationships.
• Use probabilistic observation models to account for detection failures and uncertainty in visual and language perception.
• Employ POMDP planning to generate near-optimal action sequences that include querying disambiguation questions when confidence in target identification is low.
• Optimize question generation by predicting likely human interpretations and balancing information gain against interaction cost.
• Integrate all components into a unified pipeline deployed on a Fetch robot for real-world evaluation.

Experimental results

Research questions

• RQ1 How can a robot system effectively resolve ambiguity in natural language instructions when multiple objects match the description?
• RQ2 To what extent can a POMDP framework improve robustness in visual grounding and grasping under visual occlusion and noisy perception?
• RQ3 Can learned neural networks be effectively integrated with model-based planning to reduce error propagation in complex, high-dimensional perceptual tasks?
• RQ4 How does active information gathering through disambiguation questions improve success rates in cluttered environments?
• RQ5 What is the impact of uncertainty-aware reasoning on overall system performance compared to end-to-end neural baselines?

Key findings

• INVIGORATE achieves an overall success rate of 83% on the test dataset, significantly outperforming a baseline without POMDP integration.
• Ablation studies confirm that reasoning about perceptual uncertainty and language ambiguity is critical for robust performance in cluttered scenes.
• The system successfully identifies and resolves ambiguity in language references, such as distinguishing between two remote controllers based on color cues.
• The POMDP-based planning enables effective handling of detection failures—e.g., when the target is occluded or not detected—by triggering actions like removing obstacles or asking questions.
• The critic module in question generation filters out hallucinated attributes and favors human-preferred attributes like color, improving question quality.
• Failure cases reveal limitations in OBR estimation and detection robustness, such as incorrect grasping when blocking objects are not detected or relationships are misinferred.