[Paper Review] A User's Guide to Calibrating Robotics Simulators

Bhairav Mehta, Ankur Handa+2 more|arXiv (Cornell University)|Nov 17, 2020

Reinforcement Learning in Robotics31 references4 citations

TL;DR

This paper introduces the Simulation Parameter Estimation (SIPE) benchmark to standardize evaluation of sim-to-real transfer algorithms in robotics. It evaluates parameter estimation methods using real-world trajectories, demonstrating that learned policies and discriminators improve stability and performance over manual or fixed-action approaches, with particle-based methods like ADR and SimOpt showing strong zero-shot generalization in locomotion tasks.

ABSTRACT

Simulators are a critical component of modern robotics research. Strategies for both perception and decision making can be studied in simulation first before deployed to real world systems, saving on time and costs. Despite significant progress on the development of sim-to-real algorithms, the analysis of different methods is still conducted in an ad-hoc manner, without a consistent set of tests and metrics for comparison. This paper fills this gap and proposes a set of benchmarks and a framework for the study of various algorithms aimed to transfer models and policies learnt in simulation to the real world. We conduct experiments on a wide range of well known simulated environments to characterize and offer insights into the performance of different algorithms. Our analysis can be useful for practitioners working in this area and can help make informed choices about the behavior and main properties of sim-to-real algorithms. We open-source the benchmark, training data, and trained models, which can be found at https://github.com/NVlabs/sim-parameter-estimation.

Motivation & Objective

• To address the lack of standardized benchmarks for evaluating simulation parameter estimation and sim-to-real transfer algorithms.
• To provide a unified framework for comparing system identification and calibration methods in robotics simulation.
• To analyze the impact of different trajectory generation strategies—demonstrations vs. learned policies—on parameter estimation performance.
• To evaluate the effectiveness of learned rewards versus ground-truth state differences in active domain randomization and optimization-based methods.
• To assess zero-shot generalization performance of estimated parameters in downstream reinforcement learning tasks.

Proposed method

• The SIPE benchmark provides a suite of simulated environments, expert trajectories from real robots, and trained policies for parameter estimation.
• Parameter estimation is framed as a system identification problem, where real-world trajectories are used to infer simulator parameters like mass and friction.
• The authors compare methods using both demonstrations (teleoperated or expert) and trajectories generated by trained policies, with actions derived from the policy rather than fixed.
• Active Domain Randomization (ADR) and SimOpt are evaluated with both MSE-based cost functions and learned discriminators to differentiate real vs. simulated trajectories.
• A zero-shot transfer evaluation protocol measures downstream RL agent performance using estimated parameters, normalizing results against agents trained solely in the target environment.
• The benchmark includes open-source code, datasets, and trained models for reproducibility and future research.

Experimental results

Research questions

• RQ1 How do different trajectory generation methods—demonstrations versus learned policies—affect the stability and accuracy of parameter estimation?
• RQ2 What is the impact of using learned rewards (via discriminators) versus ground-truth state differences in active domain randomization and optimization-based parameter estimation?
• RQ3 How well do particle-based methods (e.g., ADR, SimOpt) generalize to unseen environments when using estimated parameters for downstream RL training?
• RQ4 Can posterior recovery be relaxed in favor of simpler particle sampling without sacrificing downstream task performance?
• RQ5 How does the choice of cost function influence convergence and variance in parameter estimation across diverse robotic tasks?

Key findings

• Using trajectories generated by a learned policy significantly reduces variance in parameter estimation compared to fixed-action demonstrations, improving algorithm stability.
• Despite performance degradation in some environments, policy-generated trajectories stabilize learning across multiple algorithms, suggesting their value in reducing noise in the estimation process.
• The use of learned rewards via discriminators increases variance and instability, particularly in ADR, though SimOpt benefits from this approach.
• Particle-based methods like ADR and SimOpt achieve strong zero-shot generalization performance, indicating that full posterior recovery is not necessary for effective sim-to-real transfer.
• The benchmark shows that even without precise posterior estimation, sampling from converged particles leads to high downstream task performance, suggesting practical efficiency in real-world deployment.