[Paper Review] Model Identification via Physics Engines for Improved Policy Search.

Shaojun Zhu, Andrew Kimmel+2 more|arXiv (Cornell University)|Oct 24, 2017

Robot Manipulation and Learning10 citations

TL;DR

This paper proposes using off-the-shelf physics engines combined with black-box Bayesian optimization to identify mechanical parameters—such as mass and friction—in robotic systems, enabling accurate simulation-based policy search. The method improves policy learning performance for algorithms like TRPO, PoWER, and PILCO by creating a 'good enough' model that closely matches real-world trajectories, without requiring additional real-world data.

ABSTRACT

This paper presents a practical approach for identifying unknown mechanical parameters, such as mass and friction models of manipulated rigid objects or actuated robotic links, in a succinct manner that aims to improve the performance of policy search algorithms. Key features of this approach are the use of off-the-shelf physics engines and the adaptation of a black-box Bayesian optimization framework for this purpose. The physics engine is used to reproduce in simulation experiments that are performed on a real robot, and the mechanical parameters of the simulated system are automatically fine-tuned so that the simulated trajectories match with the real ones. The optimized model is then used for learning a policy in simulation, before safely deploying it on the real robot. Given the well-known limitations of physics engines in modeling real-world objects, it is generally not possible to find a mechanical model that reproduces in simulation the real trajectories exactly. Moreover, there are many scenarios where a near-optimal policy can be found without having a perfect knowledge of the system. Therefore, searching for a perfect model may not be worth the computational effort in practice. The proposed approach aims then to identify a model that is good enough to approximate the value of a locally optimal policy with a certain confidence, instead of spending all the computational resources on searching for the most accurate model. Empirical evaluations, performed in simulation and on a real robotic manipulation task, show that model identification via physics engines can significantly boost the performance of policy search algorithms that are popular in robotics, such as TRPO, PoWER and PILCO, with no additional real-world data.

Motivation & Objective

• To address the challenge of inaccurate system models in robotic policy learning due to imperfect physics engine simulations.
• To reduce the computational cost of achieving high-fidelity system identification by focusing on models that are sufficient for policy optimization.
• To enable safe and efficient policy deployment on real robots by training in simulation with calibrated models.
• To improve the performance of sample-efficient policy search algorithms like TRPO, PoWER, and PILCO through model identification.
• To demonstrate that near-optimal policies can be learned without perfect system models, provided the simulation is sufficiently close to reality.

Proposed method

• Leverage real-world robot trajectories from physical experiments as reference data for model calibration.
• Use a physics engine to simulate the same tasks and generate synthetic trajectories under varying mechanical parameters.
• Apply a black-box Bayesian optimization framework to tune unknown parameters (e.g., mass, friction) so that simulated trajectories match real ones.
• Optimize the model not for perfect accuracy but for sufficient fidelity to support learning a locally optimal policy.
• Train a policy in the calibrated simulation environment before deploying it on the real robot.
• Use the calibrated model to improve sample efficiency and performance of policy search algorithms such as TRPO, PoWER, and PILCO.

Experimental results

Research questions

• RQ1 Can physics engines be effectively used to identify mechanical parameters of robotic systems with minimal real-world data?
• RQ2 Does identifying a 'good enough' model—rather than a perfect one—lead to better policy learning performance in practice?
• RQ3 To what extent can model identification via simulation improve the performance of policy search algorithms like TRPO and PILCO?
• RQ4 How does the proposed method compare to alternative approaches in terms of computational cost and policy performance?
• RQ5 Can the calibrated simulation model reliably support safe deployment of policies on real robotic systems?

Key findings

• The proposed model identification method significantly improves the performance of policy search algorithms such as TRPO, PoWER, and PILCO in both simulation and real-world robotic manipulation tasks.
• The approach achieves better policy learning performance without requiring any additional real-world data beyond initial trajectories.
• The method successfully calibrates simulation models to closely match real-world dynamics, even when physics engines inherently lack perfect accuracy.
• The study demonstrates that a 'good enough' model—sufficient for policy optimization—can be identified with far less computational effort than achieving a perfect model.
• The calibrated simulation environment enables safe and effective policy deployment on real robots, reducing the risk of failure during real-world execution.
• Empirical results confirm that the method enhances sample efficiency and convergence speed of policy search algorithms in complex robotic manipulation tasks.