[Paper Review] Self-Supervised Policy Adaptation during Deployment

Nicklas Hansen, Rishabh Jangir+6 more|arXiv (Cornell University)|Jul 8, 2020

Reinforcement Learning in RoboticsComputer Science76 references58 citations

TL;DR

PAD enables a pre-trained policy to continue self-supervised adaptation during deployment without rewards, improving generalization to unseen environmental changes in simulation and real robots.

ABSTRACT

In most real world scenarios, a policy trained by reinforcement learning in one environment needs to be deployed in another, potentially quite different environment. However, generalization across different environments is known to be hard. A natural solution would be to keep training after deployment in the new environment, but this cannot be done if the new environment offers no reward signal. Our work explores the use of self-supervision to allow the policy to continue training after deployment without using any rewards. While previous methods explicitly anticipate changes in the new environment, we assume no prior knowledge of those changes yet still obtain significant improvements. Empirical evaluations are performed on diverse simulation environments from DeepMind Control suite and ViZDoom, as well as real robotic manipulation tasks in continuously changing environments, taking observations from an uncalibrated camera. Our method improves generalization in 31 out of 36 environments across various tasks and outperforms domain randomization on a majority of environments.

Motivation & Objective

• Motivate robust policy generalization when deploying RL agents to unseen environments.
• Propose a self-supervised adaptation signal that operates during deployment without rewards.
• Demonstrate improvement in generalization across diverse simulated and real-world tasks.

Proposed method

• Split policy network into a feature extractor and a task head to share representations.
• Use an auxiliary self-supervised task (inverse dynamics or rotation prediction) on intermediate features.
• Train with both RL objective and self-supervised objective during training; deploy with only self-supervision for adaptation.
• Online update of self-supervised head and feature extractor at test-time using new observations.

Experimental results

Research questions

• RQ1 Can self-supervision enable policy adaptation in the absence of reward signals during deployment?
• RQ2 Does online self-supervised adaptation improve generalization to unseen environmental changes in vision-based RL?
• RQ3 Which self-supervised tasks (inverse dynamics, rotation, CURL) best support deployment-time adaptation?
• RQ4 How does PAD perform across diverse simulated domains and real robotic tasks?
• RQ5 What is the impact of online vs offline learning on PAD effectiveness?

Key findings

• PAD improves generalization in 19 of 22 DMControl test environments and all CRLMaze test environments.
• PAD often outperforms domain randomization across tasks and settings.
• Inverse dynamics as self-supervision provides more consistent gains for motor-control tasks, while rotation helps scene understanding in navigation tasks.
• Online learning at deployment significantly boosts adaptation; offline variants show reduced gains.
• PAD enables Sim2Real transfer to a Kinova robot across several environments with uncalibrated camera input.