[Paper Review] LGMCTS: Language-Guided Monte-Carlo Tree Search for Executable Semantic Object Rearrangement

Haonan Chang, Kai Gao+6 more|arXiv (Cornell University)|Sep 27, 2023

Natural Language Processing TechniquesComputer Science3 citations

TL;DR

LGMCTS proposes a joint language-guided Monte Carlo Tree Search framework that concurrently generates executable semantic object rearrangement plans and poses from free-form natural language instructions. By modeling object poses as probabilistic distributions and integrating LLM-driven reasoning with MCTS, it achieves 95.99% success on line patterns and perfect 100% on tower and dinner patterns, outperforming prior two-stage methods like StructFormer and StructDiffusion in both planning and execution accuracy.

ABSTRACT

We introduce a novel approach to the executable semantic object rearrangement problem. In this challenge, a robot seeks to create an actionable plan that rearranges objects within a scene according to a pattern dictated by a natural language description. Unlike existing methods such as StructFormer and StructDiffusion, which tackle the issue in two steps by first generating poses and then leveraging a task planner for action plan formulation, our method concurrently addresses pose generation and action planning. We achieve this integration using a Language-Guided Monte-Carlo Tree Search (LGMCTS). Quantitative evaluations are provided on two simulation datasets, and complemented by qualitative tests with a real robot.

Motivation & Objective

• To address the challenge of generating not just collision-free, but truly executable, semantic object rearrangement plans from free-form natural language descriptions.
• To overcome the limitations of two-stage approaches that decouple pose generation and action planning, which often result in inexecutable or suboptimal plans.
• To enable zero-shot composition of multiple semantic patterns (e.g., line, circle, spatial) in a single rearrangement task.
• To develop a benchmark—LGR-Benchmark—specifically tailored for evaluating executable semantic rearrangement in complex, real-world-like scenarios.
• To integrate LLMs for robust language understanding and plan generation while ensuring physical feasibility through Monte Carlo Tree Search.

Proposed method

• LGMCTS uses a Large Language Model (LLM) to interpret free-form language instructions and generate probabilistic distributions over object poses, rather than fixed target positions.
• It employs Monte Carlo Tree Search (MCTS) to explore sequences of actions that simultaneously satisfy semantic constraints and physical feasibility.
• The search process models object placements as stochastic samples from LLM-predicted distributions, enabling exploration of diverse, executable configurations.
• A language-guided tree policy guides MCTS by prioritizing node expansions based on semantic alignment with the instruction and geometric plausibility.
• The planner integrates a learned collision checker to filter infeasible actions and ensure only physically executable plans are considered.
• The system is evaluated using a new benchmark, LGR-Benchmark, which includes complex, multi-pattern tasks and infeasible initial configurations.

Figure 1 : Robot Setup. We use a UR5e robot equipped with a RealSense D455 camera.

Experimental results

Research questions

• RQ1 Can a joint planning and pose generation framework outperform two-stage methods in generating executable semantic rearrangement plans?
• RQ2 How well does LGMCTS generalize to zero-shot composition of multiple semantic patterns (e.g., line + circle) in a single task?
• RQ3 To what extent does language-guided MCTS improve plan executability compared to collision-free but inexecutable plans?
• RQ4 How stable is the LLM-based instruction parsing module in real-world, open-vocabulary settings?
• RQ5 Can LGMCTS achieve high success rates in real-world robotic execution despite complex initial configurations and occlusions?

Key findings

• LGMCTS achieved a 95.99% success rate on the 'line' pattern, 95.25% on 'circle', and 100% on both 'tower' and 'dinner' patterns in simulation, significantly outperforming StructFormer and StructDiffusion.
• On the LGR-Benchmark, LGMCTS-T achieved a 97.3% planning success rate and 93.4% execution success rate, with an average plan length of 5.99 steps, outperforming PMCTS even when given ground-truth goal poses.
• The LLM-based parsing module in LGMCTS-L achieved 89.3% accuracy in object and pattern selection, demonstrating high stability in real-world instruction interpretation.
• In physical robot experiments using a UR5e with a D455 camera, LGMCTS successfully executed complex tasks like setting a dinner table with a candle, confirming real-world applicability.
• LGMCTS demonstrated superior performance even when compared to PMCTS with ground-truth poses, proving that joint modeling of pose and action planning enhances executability.
• The framework successfully handled infeasible initial configurations—such as stacked objects—by generating plans that avoid excessive intermediate steps and maintain semantic fidelity.

Figure 2 : Illustration of infeasible start and goal configuration. In this scene, block 2 is in an infeasible start configuration. And the goals of blocks 3 and 2 are infeasible without an excessive number of intermediate steps.