[Paper Review] Correcting Robot Plans with Natural Language Feedback

Pratyusha Sharma, Balakumar Sundaralingam+6 more|arXiv (Cornell University)|Apr 11, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a method for correcting robot motion plans using natural language feedback by learning to map language corrections to residual cost functions that modify the robot's planning objective. The approach achieves 81% and 93% success rates in recovering from planner failures with one or two language corrections, respectively, generalizing across unseen environments and tasks.

ABSTRACT

When humans design cost or goal specifications for robots, they often produce specifications that are ambiguous, underspecified, or beyond planners' ability to solve. In these cases, corrections provide a valuable tool for human-in-the-loop robot control. Corrections might take the form of new goal specifications, new constraints (e.g. to avoid specific objects), or hints for planning algorithms (e.g. to visit specific waypoints). Existing correction methods (e.g. using a joystick or direct manipulation of an end effector) require full teleoperation or real-time interaction. In this paper, we explore natural language as an expressive and flexible tool for robot correction. We describe how to map from natural language sentences to transformations of cost functions. We show that these transformations enable users to correct goals, update robot motions to accommodate additional user preferences, and recover from planning errors. These corrections can be leveraged to get 81% and 93% success rates on tasks where the original planner failed, with either one or two language corrections. Our method makes it possible to compose multiple constraints and generalizes to unseen scenes, objects, and sentences in simulated environments and real-world environments.

Motivation & Objective

• Address the challenge of robot planning failures due to ambiguous, under-constrained, or incorrect goal specifications.
• Enable flexible, real-time correction of robot behavior without requiring full teleoperation or continuous user attention.
• Develop a method that generalizes to unseen scenes, objects, and natural language commands through end-to-end learning of language-to-cost mappings.
• Integrate language feedback into optimization-based planning by modeling corrections as residual cost functions.
• Provide a scalable, composable framework for combining multiple language-based constraints over time during execution.

Proposed method

• Train a neural network to predict a residual cost function from visual observations and natural language corrections, conditioned on the robot's current state and environment.
• Model language corrections as modifications to the existing cost function in an optimization-based motion planner, enabling dynamic re-planning.
• Use a combination of demonstrations and ground-truth cost maps to supervise the training of the cost prediction model.
• Integrate the predicted cost function with standard motion planner costs (e.g., collision avoidance, smoothness, joint limits) for trajectory optimization.
• Enable sequential and time-varying application of corrections by layering cost functions over time during execution.
• Leverage sim-to-real transfer by training in simulation and deploying on a real robot manipulator with zero-shot generalization to new objects and environments.

Experimental results

Research questions

• RQ1 Can natural language feedback be effectively used to correct robot motion plans when the original planner fails?
• RQ2 How well can a learned language-to-cost model generalize to unseen objects, scenes, and out-of-distribution language commands?
• RQ3 Can multiple language corrections be composed to progressively refine a robot’s behavior and improve task success?
• RQ4 To what extent can a language-based correction system reduce reliance on teleoperation or kinesthetic feedback while maintaining high performance?
• RQ5 Can a single model trained in simulation successfully correct real-world robot behaviors without fine-tuning?

Key findings

• The method achieves 81% success rate in recovering from planner failures with a single natural language correction and 93% with two corrections, improving the effective success rate from 94% to 99%.
• The model generalizes to unseen objects and novel environments, demonstrating robustness to distribution shift in both visual and linguistic inputs.
• The framework supports composable, time-varying corrections, allowing users to iteratively refine robot behavior using multiple language commands.
• The system successfully transfers from simulation to real-world tasks, including cluttered scenes and non-templated natural language, without retraining.
• Language corrections are effective even when the original planner fails due to ambiguous or under-constrained goals, enabling recovery without full teleoperation.
• The approach outperforms traditional methods like joystick or kinesthetic feedback in terms of usability and user attention requirements, while maintaining high task success.