[Paper Review] Understanding Domain Randomization for Sim-to-real Transfer

Xiaoyu Chen, Jiachen Hu+3 more|arXiv (Cornell University)|Oct 7, 2021

Reinforcement Learning in RoboticsComputer Science54 references33 citations

TL;DR

The paper provides a theoretical framework for sim-to-real transfer via domain randomization, modeling simulators as latent MDPs and deriving bounds on the sim-to-real gap under various settings.

ABSTRACT

Reinforcement learning encounters many challenges when applied directly in the real world. Sim-to-real transfer is widely used to transfer the knowledge learned from simulation to the real world. Domain randomization -- one of the most popular algorithms for sim-to-real transfer -- has been demonstrated to be effective in various tasks in robotics and autonomous driving. Despite its empirical successes, theoretical understanding on why this simple algorithm works is limited. In this paper, we propose a theoretical framework for sim-to-real transfers, in which the simulator is modeled as a set of MDPs with tunable parameters (corresponding to unknown physical parameters such as friction). We provide sharp bounds on the sim-to-real gap -- the difference between the value of policy returned by domain randomization and the value of an optimal policy for the real world. We prove that sim-to-real transfer can succeed under mild conditions without any real-world training samples. Our theory also highlights the importance of using memory (i.e., history-dependent policies) in domain randomization. Our proof is based on novel techniques that reduce the problem of bounding the sim-to-real gap to the problem of designing efficient learning algorithms for infinite-horizon MDPs, which we believe are of independent interest.

Motivation & Objective

• Formulate sim-to-real transfer as a latent MDP problem with tunable simulator parameters.
• Analyze the sim-to-real gap for domain randomization under finite and infinite simulator classes.
• Show that memory (history-dependent policies) is crucial for effective sim-to-real transfer.
• Provide a novel proof framework linking sim-to-real gaps to learning in infinite-horizon MDPs with function approximation.

Proposed method

• Model the simulator as a set of MDPs with latent parameters representing real-world factors (e.g., friction).
• Define domain randomization as sampling from a distribution over MDPs to form a latent MDP with memory requirements.
• Introduce the Domain Randomization Oracle that returns an optimal history-dependent policy for the latent MDP.
• Derive upper bounds on the sim-to-real gap in three settings: finite with separation, finite without separation, and infinite simulator class.
• Connect the base-policy construction to regret bounds in infinite-horizon average-reward MDPs with function approximation.

Experimental results

Research questions

• RQ1 When does domain randomization guarantee sublinear sim-to-real gap relative to the real-world horizon H?
• RQ2 How do finite versus infinite simulator classes affect the sim-to-real gap under domain randomization?
• RQ3 What role does memory (history dependence) play in achieving favorable sim-to-real guarantees?
• RQ4 Can a provably efficient model-based algorithm be derived for learning in infinite-horizon average-reward MDPs with general function approximation relevant to domain randomization?
• RQ5 What conditions on the simulator class ensure domain randomization remains effective without real-world training data?

Key findings

• For finite simulator classes with a separation condition, the sim-to-real gap is O(D M^3 log(MH) log^2(SMH/δ) / δ^4).
• Without separation but still finite, the sim-to-real gap is O(D sqrt(M^3 H log(MH))).
• In the infinite simulator class with smoothness near the real MDP, the gap is bounded by a term involving D, eluder dimension de, horizon H, and covering numbers of the function class, plus a Lipschitz term in ε.
• A lower bound shows that any policy can incur an Ω(sqrt(D M H)) gap in worst-case finite settings without the right conditions.
• Memory (history dependence) is essential to achieving sublinear gap and cannot be discarded in domain randomization.
• The paper presents the first provably efficient model-based algorithm for learning infinite-horizon average-reward MDPs with general function approximation, achieving a near-optimal regret bound.