[Paper Review] Robot Playing Kendama with Model-Based and Model-Free Reinforcement Learning

Shidi Li|arXiv (Cornell University)|Mar 15, 2020

Reinforcement Learning in Robotics17 references4 citations

TL;DR

This paper proposes a hybrid reinforcement learning framework combining model-based (DDP) and model-free (PoWER) methods for high-precision robot trajectory learning in tasks with discontinuous dynamics, such as Kendama. By first optimizing human-imitated trajectories via DDP and then refining them with PoWER, the framework achieves successful task completion in a challenging, high-speed manipulation task.

ABSTRACT

Several model-based and model-free methods have been proposed for the robot trajectory learning task. Both approaches have their benefits and drawbacks. They can usually complement each other. Many research works are trying to integrate some model-based and model-free methods into one algorithm and perform well in simulators or quasi-static robot tasks. Difficulties still exist when algorithms are used in particular trajectory learning tasks. In this paper, we propose a robot trajectory learning framework for precise tasks with discontinuous dynamics and high speed. The trajectories learned from the human demonstration are optimized by DDP and PoWER successively. The framework is tested on the Kendama manipulation task, which can also be difficult for humans to achieve. The results show that our approach can plan the trajectories to successfully complete the task.

Motivation & Objective

• To address the challenge of learning precise, high-speed robot trajectories in tasks with discontinuous dynamics.
• To integrate model-based and model-free reinforcement learning for improved sample efficiency and robustness.
• To enable successful trajectory learning in the complex Kendama manipulation task, which is difficult even for humans.
• To optimize human-imitated demonstrations through iterative refinement using both DDP and PoWER.

Proposed method

• Human demonstration trajectories are used as initial policy for model-based optimization via Differential Dynamic Programming (DDP).
• The DDP-optimized trajectories are then refined using the model-free PoWER algorithm to improve robustness and generalization.
• The framework alternates between model-based trajectory optimization and model-free policy improvement to balance precision and adaptability.
• The method is evaluated on the Kendama task, a high-speed, discontinuous dynamics manipulation task.
• DDP ensures smooth, optimal trajectories by leveraging a learned dynamics model.
• PoWER enhances policy performance through off-policy data collection and policy gradient updates, improving success in high-dynamic scenarios.

Experimental results

Research questions

• RQ1 Can a hybrid model-based and model-free reinforcement learning framework successfully learn high-precision robot trajectories in tasks with discontinuous dynamics?
• RQ2 How does combining DDP and PoWER improve performance compared to using either method alone in complex manipulation tasks?
• RQ3 To what extent can human-imitated demonstrations be optimized and refined to achieve success in the Kendama task?
• RQ4 Does the integration of model-based planning and model-free fine-tuning enhance robustness in high-speed, dynamic manipulation?
• RQ5 Can the framework generalize to tasks that are difficult even for humans to perform?

Key findings

• The proposed framework successfully plans trajectories that enable the robot to complete the Kendama manipulation task.
• DDP effectively optimizes initial human-imitated trajectories into smooth, high-precision paths.
• PoWER improves policy robustness and generalization, enabling successful execution in high-speed, discontinuous dynamics.
• The integration of DDP and PoWER leads to better performance than using either method in isolation.
• The framework demonstrates feasibility and effectiveness in a challenging, real-world robotic manipulation task with high dynamic complexity.