[Paper Review] Auto-Tuned Sim-to-Real Transfer

Yuqing Du, Olivia Watkins+3 more|arXiv (Cornell University)|Apr 15, 2021

Domain Adaptation and Few-Shot Learning18 references4 citations

TL;DR

This paper proposes an auto-tuning method that automatically adjusts simulator system parameters to match real-world dynamics using only raw RGB video from the real world, without requiring state estimation or reward engineering. By framing parameter tuning as a search problem and using a Search Param Model (SPM) to predict whether current parameters are too high or low, the method significantly improves sim-to-real transfer performance over naive domain randomization, especially in tasks sensitive to dynamics like peg-in-hole and cabinet opening.

ABSTRACT

Policies trained in simulation often fail when transferred to the real world due to the `reality gap' where the simulator is unable to accurately capture the dynamics and visual properties of the real world. Current approaches to tackle this problem, such as domain randomization, require prior knowledge and engineering to determine how much to randomize system parameters in order to learn a policy that is robust to sim-to-real transfer while also not being too conservative. We propose a method for automatically tuning simulator system parameters to match the real world using only raw RGB images of the real world without the need to define rewards or estimate state. Our key insight is to reframe the auto-tuning of parameters as a search problem where we iteratively shift the simulation system parameters to approach the real-world system parameters. We propose a Search Param Model (SPM) that, given a sequence of observations and actions and a set of system parameters, predicts whether the given parameters are higher or lower than the true parameters used to generate the observations. We evaluate our method on multiple robotic control tasks in both sim-to-sim and sim-to-real transfer, demonstrating significant improvement over naive domain randomization. Project videos and code at https://yuqingd.github.io/autotuned-sim2real/

Motivation & Objective

• Address the sim-to-real transfer gap in robotics by reducing reliance on manual engineering for domain randomization.
• Enable automatic calibration of simulator system parameters to match real-world dynamics using only raw RGB observations from real-world videos.
• Eliminate the need for instrumented real-world state or reward function knowledge during simulation tuning.
• Improve policy generalization and success rates in real-world deployment by aligning simulation dynamics more closely with reality.

Proposed method

• Reframe the problem of sim-to-real parameter tuning as a search problem, where the goal is to iteratively adjust simulator parameters to match real-world values.
• Propose a Search Param Model (SPM) that takes a sequence of observations and actions along with current simulation parameters and predicts whether the parameters are higher, lower, or close to the true real-world values.
• Train the SPM using contrastive learning on simulated trajectories with different parameter settings, enabling it to learn to distinguish between over- and under-estimated parameter values.
• Use the SPM’s predictions to guide a reinforcement learning policy that shifts the simulation parameters toward the real-world values through iterative updates.
• Leverage the simulator’s ability to generate large-scale, diverse data with shaped rewards to train a robust policy once the parameters are auto-tuned.
• Apply the method in both sim-to-sim and sim-to-real transfer settings, using only real-world RGB video for calibration and no ground-truth state or reward.

Experimental results

Research questions

• RQ1 Can we automatically tune simulator system parameters to match real-world dynamics using only raw RGB observations from the real world?
• RQ2 Does reformulating parameter tuning as a search problem—predicting whether parameters are too high or low—enable more accurate and robust sim-to-real transfer than naive domain randomization?
• RQ3 To what extent can a model trained on simulated data with auto-tuned parameters generalize to real-world robotic control tasks without access to real-world state or reward functions?
• RQ4 How does the method perform on tasks sensitive to dynamics, such as peg-in-hole or cabinet opening, compared to standard domain randomization?

Key findings

• The proposed method outperforms naive domain randomization in sim-to-real transfer on tasks like Walker and Ball-in-Cup, with consistent improvements in success rates.
• On the rope peg-in-hole task, the method achieved consistent movement toward the hole and occasional success, while the domain randomization baseline failed to initiate proper motion.
• For the cabinet slide task, the method successfully slid the lid open consistently, whereas the domain randomization baseline consistently overshot and failed.
• The method required less than 20% of the simulation steps used by the domain randomization baseline to achieve comparable or better performance.
• Ablation studies show the method is robust in environments like Cheetah where parameters are easy to learn, but less stable in sensitive dynamics like Ball-in-Cup, indicating sensitivity to parameter identifiability.
• The method successfully learned system parameters even when the true value was initially outside the randomization range, though convergence was more challenging in such cases.