[Paper Review] Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning

Andrew S. Morgan, Daljeet Nandha+4 more|arXiv (Cornell University)|Mar 25, 2021

Reinforcement Learning in Robotics55 references32 citations

TL;DR

This paper proposes Model Predictive Actor-Critic (MoPAC), a hybrid model-based/model-free reinforcement learning algorithm that combines model predictive control rollouts with deep actor-critic learning to accelerate skill acquisition in robotics. By using MPC for efficient, guided exploration and MFRL for robust policy optimization, MoPAC reduces environment interactions while mitigating model bias, achieving faster convergence and superior performance on simulated and real-robot manipulation tasks, including in-hand manipulation with a four-fingered hand.

ABSTRACT

Substantial advancements to model-based reinforcement learning algorithms have been impeded by the model-bias induced by the collected data, which generally hurts performance. Meanwhile, their inherent sample efficiency warrants utility for most robot applications, limiting potential damage to the robot and its environment during training. Inspired by information theoretic model predictive control and advances in deep reinforcement learning, we introduce Model Predictive Actor-Critic (MoPAC), a hybrid model-based/model-free method that combines model predictive rollouts with policy optimization as to mitigate model bias. MoPAC leverages optimal trajectories to guide policy learning, but explores via its model-free method, allowing the algorithm to learn more expressive dynamics models. This combination guarantees optimal skill learning up to an approximation error and reduces necessary physical interaction with the environment, making it suitable for real-robot training. We provide extensive results showcasing how our proposed method generally outperforms current state-of-the-art and conclude by evaluating MoPAC for learning on a physical robotic hand performing valve rotation and finger gaiting--a task that requires grasping, manipulation, and then regrasping of an object.

Motivation & Objective

• To address the sample inefficiency of model-free deep reinforcement learning (MFRL) in real-robot applications, where excessive environment interactions risk damage.
• To overcome the model bias inherent in model-based RL (MBRL), which limits performance due to exploitation of imperfect dynamics models.
• To combine the sample efficiency of MBRL with the exploration capability of MFRL for faster, more robust policy learning in complex robotic tasks.
• To develop a theoretically grounded method with performance bounds under model and value function approximation errors.
• To demonstrate the effectiveness of MoPAC on challenging real-robot in-hand manipulation tasks, such as valve rotation and finger gaiting.

Proposed method

• MoPAC integrates model predictive control (MPC) rollouts with a deep actor-critic framework, using MPC to generate high-reward trajectories for guiding policy learning.
• The method employs information-theoretic MPC based on free-energy principles, incorporating an entropy constraint to balance exploitation and exploration.
• Model predictive rollouts are used to collect synthetic experience for policy optimization, reducing reliance on real environment interactions.
• A maximum entropy objective in the actor-critic component encourages exploration, enabling the learning of more expressive dynamics models.
• The algorithm uses a learned dynamics model for rollouts and a separate policy network for action selection, with both trained via off-policy updates.
• A theoretical performance bound is derived, showing that low errors in model and value function approximation lead to near-optimal performance, even with imperfect models.

Experimental results

Research questions

• RQ1 Can combining model predictive rollouts with deep actor-critic learning reduce the number of real environment interactions required for skill acquisition in robotics?
• RQ2 How does MoPAC mitigate model bias in model-based RL while maintaining sample efficiency?
• RQ3 To what extent can MoPAC accelerate learning on complex, contact-rich manipulation tasks such as in-hand object rotation and regrasping?
• RQ4 Does the integration of MPC-based planning with MFRL exploration lead to better generalization and robustness on real-robot platforms?
• RQ5 What is the theoretical performance guarantee of MoPAC under model and value function approximation errors?

Key findings

• MoPAC outperforms state-of-the-art methods such as MBPO in simulated control tasks, achieving faster convergence with fewer environment interactions.
• On a physical four-fingered robotic hand, MoPAC learned valve rotation and finger gaiting tasks significantly faster than baselines, reducing training time by up to 50% compared to MBPO.
• For the valve rotation task, MoPAC converged in approximately 2 hours, while finger gaiting required about 5 hours, with fewer episodes needed for convergence.
• The algorithm demonstrated robustness to model inaccuracies, maintaining high performance even with imperfect dynamics models, due to the theoretical performance bound on approximation errors.
• MoPAC achieved superior sample efficiency, with reduced need for physical interaction, making it suitable for real-robot deployment where environment damage is a concern.
• Empirical results confirmed that the combination of MPC rollouts for exploitation and MFRL for exploration enabled learning of more expressive dynamics and optimal policies.