[Paper Review] Policy Transfer with Strategy Optimization

Wenhao Yu, C. Karen Liu+1 more|arXiv (Cornell University)|Oct 12, 2018

Reinforcement Learning in Robotics25 references47 citations

TL;DR

SO-CMA learns a family of policies parameterized by dynamics and then searches for the best strategy in the target environment using CMA-ES, enabling robust sim-to-real transfer across large dynamics gaps.

ABSTRACT

Computer simulation provides an automatic and safe way for training robotic control policies to achieve complex tasks such as locomotion. However, a policy trained in simulation usually does not transfer directly to the real hardware due to the differences between the two environments. Transfer learning using domain randomization is a promising approach, but it usually assumes that the target environment is close to the distribution of the training environments, thus relying heavily on accurate system identification. In this paper, we present a different approach that leverages domain randomization for transferring control policies to unknown environments. The key idea that, instead of learning a single policy in the simulation, we simultaneously learn a family of policies that exhibit different behaviors. When tested in the target environment, we directly search for the best policy in the family based on the task performance, without the need to identify the dynamic parameters. We evaluate our method on five simulated robotic control problems with different discrepancies in the training and testing environment and demonstrate that our method can overcome larger modeling errors compared to training a robust policy or an adaptive policy.

Motivation & Objective

• Motivate transferring robotic control policies learned in simulation to unknown real-world-like environments with large dynamics discrepancy.
• Develop a two-stage method that avoids explicit system identification by directly optimizing over dynamic parameters based on task performance.
• Demonstrate robustness to various realism gaps including latency, actuator modeling, and deformable end-effectors across multiple locomotion tasks.
• Compare against robust, adaptive, and UPOSI baselines to show consistent transfer performance with limited target-environment samples.

Proposed method

• Train a single neural network policy conditioned on observations and dynamic parameters mu, producing a family of policies pi_mu that cover different dynamics.
• During rollout, randomly sample mu from a predefined space and fix it for the rollout to create a strategy associated with mu.
• Treat each instantiated policy pi_mu as a strategy and form a continuous policy family parameterized by mu.
• In the target environment, optimize mu* by maximizing the target-task return J_Mt(pi_mu) using CMA-ES over the mu space.
• Use PPO to train the base policy with randomized dynamics as in prior work, enabling a compact, continuous strategy representation.
• Evaluate candidate strategies by rolling out in the target environment and computing fitness, guiding CMA-ES updates.

Experimental results

Research questions

• RQ1 Can strategy optimization over a parameterized family of policies achieve better transfer than robust, adaptive, or UPOSI baselines when the target dynamics differ substantially from training dynamics?
• RQ2 How does the dimensionality of the randomized dynamics space dim(mu) affect transfer performance and sample efficiency in the target domain?
• RQ3 Is CMA-ES an effective optimizer for searching over dynamics parameters under sparse reward conditions in the target environment?
• RQ4 What kinds of real-world-like discrepancies (latency, actuator modeling, deformable end-effectors) can SO-CMA handle during sim-to-real transfer?

Key findings

• SO-CMA consistently outperforms baseline methods (Robust, Hist, UPOSI) across several transfer scenarios with limited target samples.
• Increasing dim(mu) enables better transfer when the baseline methods struggle, and CMA-ES-based strategy optimization achieves strong end performance.
• SO-CMA succeeds in transferring policies despite large modeling errors such as latency and actuator modeling differences, where robust or adaptive policies falter.
• In cases with sparse rewards, CMA-ES-based strategy search remains effective, whereas some baselines rely on dense rewards for fine-tuning.
• The method also demonstrates transfer from rigid to deformable end-effectors, indicating versatility across challenging transfer tasks.