[Paper Review] Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model

Paul F. Christiano, Zain Shah+6 more|arXiv (Cornell University)|Oct 11, 2016

Reinforcement Learning in Robotics3 references166 citations

TL;DR

The paper presents a method to transfer policies trained in simulation to the real world by learning a deep inverse dynamics model in the target domain, using the simulator to predict next observations and adapt actions accordingly.

ABSTRACT

Developing control policies in simulation is often more practical and safer than directly running experiments in the real world. This applies to policies obtained from planning and optimization, and even more so to policies obtained from reinforcement learning, which is often very data demanding. However, a policy that succeeds in simulation often doesn't work when deployed on a real robot. Nevertheless, often the overall gist of what the policy does in simulation remains valid in the real world. In this paper we investigate such settings, where the sequence of states traversed in simulation remains reasonable for the real world, even if the details of the controls are not, as could be the case when the key differences lie in detailed friction, contact, mass and geometry properties. During execution, at each time step our approach computes what the simulation-based control policy would do, but then, rather than executing these controls on the real robot, our approach computes what the simulation expects the resulting next state(s) will be, and then relies on a learned deep inverse dynamics model to decide which real-world action is most suitable to achieve those next states. Deep models are only as good as their training data, and we also propose an approach for data collection to (incrementally) learn the deep inverse dynamics model. Our experiments shows our approach compares favorably with various baselines that have been developed for dealing with simulation to real world model discrepancy, including output error control and Gaussian dynamics adaptation.

Motivation & Objective

• Leverage a competent source-domain policy to perform well in a target domain (often real) despite simulation-reality discrepancies.
• Exploit the idea that high-level policy behavior transfers, while low-level control details differ due to friction, contacts, and other dynamics.
• Develop an online data-collection strategy to train a deep inverse dynamics model that adapts actions in the target domain.
• Demonstrate transfer efficacy through Sim1→Sim2 and Sim→Real experiments, including contact-rich tasks.
• Compare against baselines that address model mismatch via output error control or Gaussian dynamics adaptation.

Proposed method

• At each time step, compute a source-domain action a_source = pi_source(tau_-k:).
• Predict the next source-domain observation o_next_hat = o(T_source(tau_-k:, a_source)).
• Use a learned inverse dynamics model phi(tau_-k:, o_next_hat) to select the target-domain action a_target.
• Train phi to map (oHistory, aHistory, o_next) to the preceding action that achieves the transition.
• Incorporate history window H to capture temporal dependencies and latent factors in dynamics.
• Collect training data by executing a preliminary target-domain policy with selective exploration noise and iteratively refining phi.

Experimental results

Research questions

• RQ1 Can a deep inverse dynamics model learned in the target domain enable effective transfer from a source-domain policy to the target domain?
• RQ2 Does using predicted next observations and an inverse model outperform direct policy transfer or forward-dynamics adaptation in simulation-to-real transfer, especially with contact-rich dynamics?
• RQ3 How does history-aware inverse dynamics learning impact data efficiency and adaptation performance?
• RQ4 What is the comparative performance of the proposed method against output error control and Gaussian dynamics adaptation baselines in varied dynamics?
• RQ5 Is action adaptation sufficient to achieve robust Sim-to-Real transfer without state/observation adaptation?

Key findings

• The proposed method achieves compelling transfer from simulation to real world, including challenging contact-rich dynamics.
• Adaptation outperforms baseline methods (output error control and Gaussian dynamics adaptation) in both Sim1→Sim2 and Sim→Real settings.
• Using history in the inverse dynamics model reduces data requirements and improves convergence.
• Learning with targeted, task-relevant data collection yields faster convergence than random exploration.
• In Sim→Real Fetch experiments, the method significantly reduces deviation from the simulated trajectory compared with a PD baseline.
• The approach remains effective across variations in gravity and motor noise, and handles discontinuities arising from contacts.