[Paper Review] DARLA: Improving Zero-Shot Transfer in Reinforcement Learning

Irina Higgins, Arka Pal+7 more|arXiv (Cornell University)|Jul 26, 2017

Reinforcement Learning in RoboticsComputer Science49 references139 citations

TL;DR

DARLA learns disentangled latent representations from unsupervised vision in stage one, then trains a source policy in stage two, enabling zero-shot transfer across diverse domain shifts without target-domain data. This approach improves robustness across DeepMind Lab and Jaco/MuJoCo environments and multiple RL algorithms.

ABSTRACT

Domain adaptation is an important open problem in deep reinforcement learning (RL). In many scenarios of interest data is hard to obtain, so agents may learn a source policy in a setting where data is readily available, with the hope that it generalises well to the target domain. We propose a new multi-stage RL agent, DARLA (DisentAngled Representation Learning Agent), which learns to see before learning to act. DARLA's vision is based on learning a disentangled representation of the observed environment. Once DARLA can see, it is able to acquire source policies that are robust to many domain shifts - even with no access to the target domain. DARLA significantly outperforms conventional baselines in zero-shot domain adaptation scenarios, an effect that holds across a variety of RL environments (Jaco arm, DeepMind Lab) and base RL algorithms (DQN, A3C and EC).

Motivation & Objective

• Motivate domain adaptation in RL where target-domain data is scarce or unavailable.
• Propose a three-stage pipeline (learn to see, learn to act, transfer) to achieve robust zero-shot transfer.
• Show that disentangled representations lead to policies robust to domain shifts across multiple environments and RL algorithms.
• Demonstrate the limitations of entangled representations and baselines in zero-shot transfer.
• Provide empirical evidence across to-be-tested environments to support robustness claims.

Proposed method

• Three-stage DARLA pipeline: 1) Learn to see with unsupervised disentangled representation learning (F_U) from raw observations; 2) Learn to act using the learned latent state s^z with a source-domain policy; 3) Transfer by evaluating zero-shot transfer to a target domain without retraining.
• Use beta-VAE (and a beta-VAE with perceptual similarity loss called beta-VAE_DAE) to learn disentangled latent factors that correspond to high-level generative factors of variation.
• Keep the vision module F_U fixed during the RL stage to force transfer-friendly representations.
• Apply standard RL algorithms (DQN, A3C, Episodic Control) on the latent state to learn the source policy.
• Compare disentangled representation DARLA to baselines with entangled representations and to denoising autoencoder baselines (DARLA_ENT, DARLA_DAE).
• Evaluate on diverse tasks: DeepMind Lab and Jaco/MuJoCo in sim2sim and sim2real scenarios to assess robustness to domain shifts.

Experimental results

Research questions

• RQ1 Does learning a disentangled latent representation in an unsupervised stage improve zero-shot transfer in RL across domain shifts?
• RQ2 How does DARLA desempenho compare to baselines with entangled representations or alternative unsupervised vision models in various RL environments?
• RQ3 To what extent is the transfer performance correlated with the degree of disentanglement of the learned representation?
• RQ4 Can the approach handle both domain interpolation and extrapolation, including perceptual reality gaps (sim2real)?

Key findings

• DARLA significantly outperforms conventional baselines in zero-shot domain adaptation across multiple environments and base RL algorithms.
• Disentangled latent representations learned in stage one enable the policy to ignore irrelevant domain attributes, improving generalisation to target domains.
• DARLA shows robust zero-shot transfer in both DeepMind Lab and Jaco/MuJoCo tasks and across DQN, A3C, and Episodic Control.
• There is a positive correlation between disentanglement level and transfer performance (transfer/disentanglement score, r = 0.6, p < 0.001 in one reported analysis).
• The improvement is demonstrated as median improvements (e.g., reported as 270.3% in one context) in zero-shot transfer, indicating substantial robustness gains.
• Learning good vision (disentangled representations) is crucial; entangled representations perform worse in transfer.