[Paper Review] Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation

Shani Gamrian, Yoav Goldberg|arXiv (Cornell University)|May 31, 2018

Generative Adversarial Networks and Image SynthesisComputer Science36 references40 citations

TL;DR

The paper shows that separating visual transfer from control policies via unaligned GANs enables effective zero-shot transfer and improved sample efficiency for related RL tasks, demonstrated on Breakout variants and Road Fighter, with imitation learning from imperfect demonstrations enhancing results.

ABSTRACT

Despite the remarkable success of Deep RL in learning control policies from raw pixels, the resulting models do not generalize. We demonstrate that a trained agent fails completely when facing small visual changes, and that fine-tuning---the common transfer learning paradigm---fails to adapt to these changes, to the extent that it is faster to re-train the model from scratch. We show that by separating the visual transfer task from the control policy we achieve substantially better sample efficiency and transfer behavior, allowing an agent trained on the source task to transfer well to the target tasks. The visual mapping from the target to the source domain is performed using unaligned GANs, resulting in a control policy that can be further improved using imitation learning from imperfect demonstrations. We demonstrate the approach on synthetic visual variants of the Breakout game, as well as on transfer between subsequent levels of Road Fighter, a Nintendo car-driving game. A visualization of our approach can be seen in https://youtu.be/4mnkzYyXMn4 and https://youtu.be/KCGTrQi6Ogo .

Motivation & Objective

• Demonstrate that standard deep RL fails to generalize to small visual changes and that fine-tuning transfers poorly.
• Propose separating visual transfer from policy learning to enable effective transfer across visually different but dynamically related tasks.
• Show that unaligned GANs can map target-domain visuals to source-domain visuals to reuse learned policies.
• Propose an imitation learning framework to improve transferred policies when GAN mappings are imperfect.

Proposed method

• Train a source RL agent on a base task (e.g., Breakout) and test on visually altered target tasks.
• Attempt standard transfer via fine-tuning with various frozen/fresh layer configurations; observe failure to transfer.
• Learn a visual mapping G from target to source domain using unaligned GANs (UNIT/CycleGAN framework) without paired data.
• Evaluate transferred policy π(a|G(t); θ) by interaction in the target environment using the source policy parameters θ.
• When GAN mappings are imperfect, treat the transferred policy as imperfect demonstrations and apply imitation learning to improve sample efficiency.
• Optionally combine imitation-learning updates with on-policy RL (A2C) updates according to algorithmic details in the paper.

Experimental results

Research questions

• RQ1 Can deep RL policies trained on a source task generalize to visually altered target tasks with minimal dynamics change?
• RQ2 Is fine-tuning an effective transfer method for visually variant RL tasks, or does it hinder learning?
• RQ3 Can an agent transfer knowledge across different levels/variants by separating visual transfer from control dynamics?
• RQ4 Does a zero-shot visual transfer via unaligned GANs enable usable policies, and can imitation learning improve such policies when GANs are imperfect?

Key findings

• Fine-tuning largely fails to transfer between Breakout variants, sometimes converging as slowly as training from scratch or performing worse.
• Transferring between Road Fighter levels without visual transfer yields zero score, indicating complete policy failure.
• Zero-shot visual transfer using unaligned GANs (G maps target visuals to source visuals) can enable high target-task scores with far fewer target interactions (sample-efficient transfer).
• Among GAN choices, UNIT-based translation often outperforms CycleGAN on Breakout, while results on Road Fighter are more mixed; GAN quality directly impacts policy performance.
• Imitation learning from imperfect GAN-based demonstrations further improves target-task performance and sample efficiency, enabling achievement of high scores with significantly fewer environment interactions than training from scratch.
• The approach also provides a practical, task-oriented metric to evaluate unaligned GANs based on downstream RL performance.