[Paper Review] Model-based Reinforcement Learning with Parametrized Physical Models and Optimism-Driven Exploration

Christopher Xie, Sachin Patil+3 more|arXiv (Cornell University)|Sep 23, 2015

Reinforcement Learning in Robotics14 references3 citations

TL;DR

This paper proposes a model-based reinforcement learning method that combines parametrized physical dynamics models with optimism-driven exploration via model predictive control (MPC). By leveraging morphology-derived features for fast least-squares dynamics identification and using optimistic MPC for goal-directed exploration, the approach achieves state-of-the-art sample efficiency and real-time performance on benchmark robotic tasks, including a 7-DOF arm, outperforming prior methods in interaction efficiency and computation speed.

ABSTRACT

In this paper, we present a robotic model-based reinforcement learning method that combines ideas from model identification and model predictive control. We use a feature-based representation of the dynamics that allows the dynamics model to be fitted with a simple least squares procedure, and the features are identified from a high-level specification of the robot's morphology, consisting of the number and connectivity structure of its links. Model predictive control is then used to choose the actions under an optimistic model of the dynamics, which produces an efficient and goal-directed exploration strategy. We present real time experimental results on standard benchmark problems involving the pendulum, cartpole, and double pendulum systems. Experiments indicate that our method is able to learn a range of benchmark tasks substantially faster than the previous best methods. To evaluate our approach on a realistic robotic control task, we also demonstrate real time control of a simulated 7 degree of freedom arm.

Motivation & Objective

• To accelerate learning in model-based reinforcement learning by integrating domain knowledge of robot morphology into dynamics modeling.
• To address the inefficiency of general-purpose statistical models in continuous control tasks requiring many interaction samples.
• To enable real-time online learning by reducing computational overhead in model identification and exploration.
• To develop a method that combines fast least-squares dynamics fitting with optimism-driven exploration for efficient task completion.
• To demonstrate applicability on complex, high-dimensional robotic systems such as a 7-DOF arm in simulation.

Proposed method

• Uses a feature-based linear dynamics model derived from high-level robot morphology (number and connectivity of links), enabling fast least-squares fitting.
• Applies model predictive control (MPC) with optimism in the face of uncertainty to guide exploration toward goal-directed trajectories.
• Reduces optimism over time as more interaction data is collected, allowing convergence to the true dynamics.
• Automatically generates dynamics features from robot structure using tools like SymPyBotics, avoiding manual feature engineering.
• Employs a parametrized dynamics model where the state transition is expressed as a linear combination of learned features.
• Integrates online model identification with MPC-based control, enabling real-time adaptation during task execution.

Experimental results

Research questions

• RQ1 Can incorporating physical prior knowledge into dynamics modeling significantly reduce sample complexity in model-based RL for robotic control?
• RQ2 Can optimism-driven exploration via MPC be made computationally efficient enough for real-time online learning on high-dimensional robots?
• RQ3 How does the proposed method compare in sample efficiency and computation time to prior model-based RL approaches on standard benchmarks?
• RQ4 Can the method generalize to complex, high-DOF robotic systems such as a 7-DOF arm with unknown dynamics?
• RQ5 To what extent does the method remain robust to unmodeled dynamics or real-world perturbations?

Key findings

• The method achieved the lowest interaction time among all compared methods on pendulum, cartpole, and double pendulum benchmarks, demonstrating superior sample efficiency.
• On the 7-DOF arm task, the method successfully reached target poses in all 10 trials, with interaction times only 2–3 times slower than DDP with known dynamics.
• Computation time for each control step was comparable to interaction time, confirming real-time feasibility for online learning.
• The dynamics model for the 7-DOF arm had 70 parameters, significantly more complex than the benchmarks (<10 parameters), yet the method remained efficient.
• The approach outperformed a prior MPC-based optimistic method [21] in both sample efficiency and computational speed, enabling real-time deployment.
• The method demonstrated that using morphology-based features reduces the need for extensive data collection, even in high-dimensional systems.