[Paper Review] Inner Monologue: Embodied Reasoning through Planning with Language Models

Wenlong Huang, Fei Xia+15 more|arXiv (Cornell University)|Jul 12, 2022

Multimodal Machine Learning Applications206 citations

TL;DR

The paper shows that feeding grounded environment feedback into a frozen language model enables an inner-monologue planning loop that improves long-horizon robotic tasks in simulation and real-world settings, without additional training.

ABSTRACT

Recent works have shown how the reasoning capabilities of Large Language Models (LLMs) can be applied to domains beyond natural language processing, such as planning and interaction for robots. These embodied problems require an agent to understand many semantic aspects of the world: the repertoire of skills available, how these skills influence the world, and how changes to the world map back to the language. LLMs planning in embodied environments need to consider not just what skills to do, but also how and when to do them - answers that change over time in response to the agent's own choices. In this work, we investigate to what extent LLMs used in such embodied contexts can reason over sources of feedback provided through natural language, without any additional training. We propose that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan in robotic control scenarios. We investigate a variety of sources of feedback, such as success detection, scene description, and human interaction. We find that closed-loop language feedback significantly improves high-level instruction completion on three domains, including simulated and real table top rearrangement tasks and long-horizon mobile manipulation tasks in a kitchen environment in the real world.

Motivation & Objective

• Investigate whether natural language feedback from embodied environments can be leveraged by frozen LLM planners to improve planning and execution in robotics.
• Develop and analyze the Inner Monologue framework that interleaves perception feedback, robot skills, and human feedback within LLM-driven planning.
• Evaluate the approach across simulated tabletop rearrangement, real-world tabletop rearrangement, and real-world kitchen mobile manipulation to assess robustness and generalization.
• Explore emergent capabilities of the inner-monologue approach, such as adaptation to new instructions, self-proposed goals, multilingual interaction, and interactive scene understanding.

Proposed method

• Propose Inner Monologue: inject environment-derived textual feedback (success signals, scene descriptions, and human input) into the LLM planner during planning and execution.
• Use pre-trained language-conditioned robotic skills as the action library for the planner to assemble into plans.
• Ground the feedback via perception modules (object recognition, scene description, success detectors) and, in some setups, affordance-based grounding for real robots.
• Maintain a few-shot prompting regime with a frozen LLM, not fine-tuned on task data.
• Demonstrate closed-loop interaction where the LLM can retry, replan, or request human feedback in response to feedback from the environment.
• Evaluate across three domains: simulated tabletop rearrangement, real-world tabletop rearrangement, and real-world kitchen mobile manipulation.

Experimental results

Research questions

• RQ1 Can an LLM planner meaningfully utilize textual environment feedback to close the loop with perception and control in embodied tasks?
• RQ2 Does inner-monologue planning improve success on long-horizon manipulation tasks compared to open-loop or non-LLM baselines?
• RQ3 What types of feedback (success signals, passive scene descriptions, active scene descriptions, human input) best support planning and execution?
• RQ4 What emergent capabilities arise from grounding LLM planning in embodied feedback (e.g., adaptation, multilingual interaction, interactive scene understanding)?

Key findings

• Inner Monologue variants that incorporate object recognition and scene feedback achieve higher task success than baselines across simulated and real tasks.
• Object + Scene feedback with chain-of-thought prompts yields strong generalization to unseen tasks in simulation.
• In real-world tabletop tasks, adding success feedback alongside object recognition significantly improves robustness and recovery from failures.
• In real-world kitchen manipulation, LLM-informed feedback enables better retry and replanning under disturbances, outperforming SayCan baselines in challenged scenarios.
• Emergent capabilities include continued adaptation to new instructions, self-proposing goals under infeasibility, multilingual interaction, and interactive scene understanding, even without explicit prompting for these behaviors.