[Paper Review] REvolveR: Continuous Evolutionary Models for Robot-to-robot Policy Transfer

Xingyu Liu, Deepak Pathak+1 more|arXiv (Cornell University)|Feb 10, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

REvolveR proposes a continuous evolutionary model for robot-to-robot policy transfer by gradually evolving robot morphology and kinematics from a source to a target robot in a physics simulator. By training a policy on intermediate, smoothly transformed robots, the method achieves superior sample efficiency and succeeds in sparse reward settings where imitation learning fails.

ABSTRACT

A popular paradigm in robotic learning is to train a policy from scratch for every new robot. This is not only inefficient but also often impractical for complex robots. In this work, we consider the problem of transferring a policy across two different robots with significantly different parameters such as kinematics and morphology. Existing approaches that train a new policy by matching the action or state transition distribution, including imitation learning methods, fail due to optimal action and/or state distribution being mismatched in different robots. In this paper, we propose a novel method named $REvolveR$ of using continuous evolutionary models for robotic policy transfer implemented in a physics simulator. We interpolate between the source robot and the target robot by finding a continuous evolutionary change of robot parameters. An expert policy on the source robot is transferred through training on a sequence of intermediate robots that gradually evolve into the target robot. Experiments on a physics simulator show that the proposed continuous evolutionary model can effectively transfer the policy across robots and achieve superior sample efficiency on new robots. The proposed method is especially advantageous in sparse reward settings where exploration can be significantly reduced. Code is released at https://github.com/xingyul/revolver.

Motivation & Objective

• Address the challenge of transferring policies between robots with significantly different morphologies and dynamics, where traditional imitation learning fails due to mismatched optimal action/state distributions.
• Overcome the inefficiency and impracticality of retraining policies from scratch for each new robot, especially in complex or high-dimensional environments.
• Enable effective policy transfer in sparse reward settings where exploration is difficult and success signals are rare.
• Develop a scalable framework that decomposes difficult inter-robot transfer into a sequence of easier, incremental policy fine-tuning tasks.
• Improve training stability and sample efficiency through novel components like local randomized evolution and evolution reward shaping.

Proposed method

• Define a continuous evolutionary model that interpolates between source and target robots by smoothly varying parameters such as leg length, mass, and finger configuration in a physics simulator.
• Train the policy on a sequence of intermediate robots that gradually evolve toward the target robot, enabling incremental adaptation to changing dynamics.
• Implement a local randomized evolution strategy where each training epoch samples a random subset of robots within a small range of evolution parameters to improve robustness and generalization.
• Introduce evolution reward shaping that assigns higher rewards to transitions on more evolved robots, encouraging faster convergence toward the target robot.
• Use a differentiable interpolation scheme to represent robot parameters in continuous space, enabling smooth transitions and gradient-based optimization.
• Integrate the method with standard RL algorithms (e.g., NPG, SAC) and apply adaptive training scheduling to enhance sample efficiency.

Experimental results

Research questions

• RQ1 Can a policy trained on a source robot be effectively transferred to a target robot with significantly different morphology and dynamics using a continuous evolutionary framework?
• RQ2 How does local randomized evolution improve training stability and robustness during policy transfer across evolving robots?
• RQ3 To what extent does evolution reward shaping enhance sample efficiency and convergence speed in policy transfer?
• RQ4 Can the proposed method achieve successful policy transfer in sparse reward environments where standard imitation learning and direct transfer fail?
• RQ5 How does the continuous evolutionary model compare to direct transfer and imitation learning baselines in terms of sample efficiency and final performance?

Key findings

• REvolveR outperforms direct transfer and imitation learning baselines in both dense and sparse reward settings, achieving higher sample efficiency and faster convergence.
• In sparse reward settings, REvolveR achieved 90% success rate in 6,279.35 ± 290.33 optimization steps with evolution reward shaping (h=1.0), significantly outperforming baselines that failed to learn.
• The use of local randomized evolution reduced standard deviation in episode rewards, indicating improved training robustness compared to deterministic evolution.
• Even with sparse rewards, REvolveR required fewer optimization steps than baselines trained with dense rewards, demonstrating its superior sample efficiency.
• Ablation studies confirmed that both local randomized evolution and evolution reward shaping independently improve performance, with the combination yielding the best results.
• The method successfully transferred policies from a five-finger robot to a two-finger gripper in complex manipulation tasks (Hammer, Relocate, Door), where human demonstration-based methods failed entirely.