[Paper Review] DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment

Yanjiang Guo, Yen‐Jen Wang+2 more|arXiv (Cornell University)|Jul 1, 2023

Topic Modeling4 citations

TL;DR

DoReMi is a novel framework that enables large language models (LLMs) to detect and recover from plan-execution misalignments in robotic tasks by using LLMs to generate execution constraints and vision-language models (VLMs) to continuously monitor for constraint violations. Experiments show it improves task success rates and reduces completion time in robot arm and humanoid robot tasks compared to baseline methods.

ABSTRACT

Large language models (LLMs) encode a vast amount of semantic knowledge and possess remarkable understanding and reasoning capabilities. Previous work has explored how to ground LLMs in robotic tasks to generate feasible and executable textual plans. However, low-level execution in the physical world may deviate from the high-level textual plan due to environmental perturbations or imperfect controller design. In this paper, we propose extbf{DoReMi}, a novel language model grounding framework that enables immediate Detection and Recovery from Misalignments between plan and execution. Specifically, we leverage LLMs to play a dual role, aiding not only in high-level planning but also generating constraints that can indicate misalignment during execution. Then vision language models (VLMs) are utilized to detect constraint violations continuously. Our pipeline can monitor the low-level execution and enable timely recovery if certain plan-execution misalignment occurs. Experiments on various complex tasks including robot arms and humanoid robots demonstrate that our method can lead to higher task success rates and shorter task completion times. Videos of DoReMi are available at \url{https://sites.google.com/view/doremi-paper}.

Motivation & Objective

• To address the critical gap in robotic task execution where low-level actions deviate from high-level language plans due to environmental disturbances or controller errors.
• To enable immediate detection of such misalignments during execution, rather than relying on delayed feedback after step completion.
• To improve task success rates and reduce execution time by allowing rapid re-planning upon detecting constraint violations.
• To leverage the dual role of LLMs in both high-level planning and generating executable constraints for low-level monitoring.
• To integrate vision-language models as lightweight, precise constraint detectors that focus only on relevant physical conditions.

Proposed method

• LLMs generate high-level textual plans with embedded physical constraints (e.g., 'the red block is on the brown block') to guide execution.
• During execution, a vision-language model (VLM) continuously monitors the environment and checks whether any constraints are violated.
• If a constraint violation is detected (e.g., the red block is no longer on the brown block), the system triggers immediate re-planning via the LLM.
• The re-planning process dynamically adjusts the action sequence based on the current state and updated constraints.
• The framework operates in a closed-loop manner, enabling continuous monitoring and recovery throughout task execution.
• The method relies on LLMs for constraint generation and VLMs for real-time, binary constraint validation, minimizing computational overhead.

Experimental results

Research questions

• RQ1 Can real-time detection of plan-execution misalignment improve task success rates in long-horizon robotic tasks?
• RQ2 How does immediate re-planning upon constraint violation compare to delayed feedback in terms of task completion time?
• RQ3 Can LLMs effectively generate meaningful, executable constraints that reflect physical world dependencies?
• RQ4 To what extent can VLMs accurately and efficiently detect constraint violations during low-level robot execution?
• RQ5 What is the theoretical and empirical improvement in task performance when using immediate recovery from misalignment?

Key findings

• DoReMi significantly increases task success rates in complex manipulation tasks involving robot arms and humanoid robots.
• The framework reduces average task completion time by enabling immediate recovery instead of continuing with failed actions.
• Constraint violations are detected in real time by the VLM, allowing prompt re-planning without waiting for step completion.
• The dual use of LLMs for planning and constraint generation improves alignment between high-level intent and low-level execution.
• The integration of VLMs as focused constraint detectors provides precise feedback with minimal computational cost.
• Empirical results in simulation demonstrate that DoReMi outperforms baseline methods that rely on delayed feedback or lack constraint monitoring.