[Paper Review] Planning with a Receding Horizon for Manipulation in Clutter using a Learned Value Function

Wissam Bejjani, Rafael Papallas+2 more|White Rose Research Online (University of Leeds, The University of Sheffield, University of York)|Mar 21, 2018

Reinforcement Learning in Robotics17 references4 citations

TL;DR

This paper proposes a receding horizon planner (RHP) with a learned value function for real-time, closed-loop manipulation in cluttered environments. By combining a sampling-based planner to generate initial plans and fine-tuning the value function via reinforcement learning, the method enables fast, reactive planning that significantly outperforms open-loop approaches under real-world uncertainty, achieving high success rates even with physics parameter mismatches.

ABSTRACT

Manipulation in clutter requires solving complex sequential decision making problems in an environment rich with physical interactions. The transfer of motion planning solutions from simulation to the real world, in open-loop, suffers from the inherent uncertainty in modelling real world physics. We propose interleaving planning and execution in real-time, in a closed-loop setting, using a Receding Horizon Planner (RHP) for pushing manipulation in clutter. In this context, we address the problem of finding a suitable value function based heuristic for efficient planning, and for estimating the cost-to-go from the horizon to the goal. We estimate such a value function first by using plans generated by an existing sampling-based planner. Then, we further optimize the value function through reinforcement learning. We evaluate our approach and compare it to state-of-the-art planning techniques for manipulation in clutter. We conduct experiments in simulation with artificially injected uncertainty on the physics parameters, as well as in real world tasks of manipulation in clutter. We show that this approach enables the robot to react to the uncertain dynamics of the real world effectively.

Motivation & Objective

• Address the challenge of executing motion plans in cluttered, physics-based environments where real-world dynamics deviate from simulation models.
• Overcome the limitations of open-loop planning, which fails under physical modeling inaccuracies and contact-rich interactions.
• Develop a fast, closed-loop planning system that reacts to environmental uncertainty in real time.
• Learn a robust, data-driven value function to estimate cost-to-go for efficient receding horizon planning without manual heuristic design.
• Improve planning performance through end-to-end reinforcement learning fine-tuning of the value function after initial pre-training.

Proposed method

• Use a sampling-based kino-dynamic RRT planner to generate expert demonstrations in simulation, creating state-action-value pairs for pre-training.
• Train a deep neural network (DNN) to predict the value of a state-action pair, estimating the expected cost-to-go from that state.
• Implement a receding horizon planner (RHP) that plans over a short time horizon, using the DNN-predicted value function as a heuristic to guide action selection.
• Fine-tune the DNN value function using reinforcement learning (PPO) to align predictions with actual RHP execution outcomes, improving robustness.
• Integrate real-time feedback by re-planning at each step using updated observations from the environment, enabling closed-loop adaptation.
• Use Box2D physics simulation with injected Gaussian noise on object parameters (shape, friction, density) to simulate real-world uncertainty during evaluation.

Experimental results

Research questions

• RQ1 Can a learned value function improve the efficiency and robustness of receding horizon planning in cluttered manipulation tasks?
• RQ2 How does fine-tuning a pre-trained value function via reinforcement learning affect performance under physical modeling uncertainty?
• RQ3 To what extent does closed-loop RHP with a learned value function outperform open-loop planning in real-world execution scenarios?
• RQ4 Can a data-driven value function eliminate the need for hand-engineered heuristics or reward shaping in complex manipulation tasks?
• RQ5 How does the choice of planning horizon and roll-out count impact the trade-off between success rate and computation time in uncertain environments?

Key findings

• The RHP with a learned value function (RHP-66) achieved a 96% success rate under high physics parameter uncertainty (30% deviation), significantly outperforming the open-loop kino-dynamic planner, which dropped to 44% under the same conditions.
• RHP with a horizon of 6 and 66 roll-outs (RHP-66) achieved the highest success rate (96%) across all uncertainty levels, demonstrating the effectiveness of longer horizons and sufficient roll-outs.
• Computation time per successful execution remained within reasonable limits (under 2 seconds on average), enabling near real-time operation despite the use of reinforcement learning.
• The real-world experiments on a UR5 robot confirmed the method’s robustness: RHP succeeded in all three tested tasks, while the open-loop planner failed in two out of three.
• The combination of pre-training on sampling-based plans and fine-tuning via PPO led to a value function that generalized better to real-world dynamics than the initial DNN alone.
• The system demonstrated strong adaptability to unexpected physical behaviors, such as object shape mismatches, by continuously re-planning based on real-time observations.