[论文解读] Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design

Michael D. Dennis, Natasha Jaques+5 more|arXiv (Cornell University)|Dec 3, 2020

Reinforcement Learning in Robotics参考文献 51被引用 40

一句话总结

tldr: 引入 Unsupervised Environment Design (UED) 和 PAIRED 算法，自动生成结构化、可解决的环境，通过一个依赖学习进度的对手对主角代理进行课程式学习，从而实现更好的零样本迁移和涌现复杂性。

ABSTRACT

A wide range of reinforcement learning (RL) problems - including robustness, transfer learning, unsupervised RL, and emergent complexity - require specifying a distribution of tasks or environments in which a policy will be trained. However, creating a useful distribution of environments is error prone, and takes a significant amount of developer time and effort. We propose Unsupervised Environment Design (UED) as an alternative paradigm, where developers provide environments with unknown parameters, and these parameters are used to automatically produce a distribution over valid, solvable environments. Existing approaches to automatically generating environments suffer from common failure modes: domain randomization cannot generate structure or adapt the difficulty of the environment to the agent's learning progress, and minimax adversarial training leads to worst-case environments that are often unsolvable. To generate structured, solvable environments for our protagonist agent, we introduce a second, antagonist agent that is allied with the environment-generating adversary. The adversary is motivated to generate environments which maximize regret, defined as the difference between the protagonist and antagonist agent's return. We call our technique Protagonist Antagonist Induced Regret Environment Design (PAIRED). Our experiments demonstrate that PAIRED produces a natural curriculum of increasingly complex environments, and PAIRED agents achieve higher zero-shot transfer performance when tested in highly novel environments.

研究动机与目标

• Formalize the problem of Unsupervised Environment Design (UED) and its relation to decisions under ignorance.
• Introduce the Protagonist Antagonist Induced Regret Environment Design (PAIRED) algorithm.
• Demonstrate that UED with PAIRED yields emergent complexity and improved zero-shot transfer.
• Connect UED to classical decision theory and provide theoretical properties under regret.
• Provide empirical evidence that PAIRED outperforms domain randomization and minimax adversarial approaches in transfer scenarios.

提出的方法

• Model fully specified environments as POMDPs and underspecified environments (UPOMDPs) with free parameters Θ.
• Define environment policies that map policies to distributions over Θ trajectories to generate environment samples.
• Introduce regret as the difference between antagonist and protagonist rewards in the same environment setting.
• Train protagonist, antagonist, and environment-adversary to maximize/minimize regret via RL updates (Algorithm 1).
• Use multiple trajectories per environment to better approximate regret and guide adversary curricula.
• Prove theoretical properties: minimax regret yields policies that succeed when success is possible (Theorem 1) and that Nash equilibria correspond to minimax regret policies (Theorem 2).

实验结果

研究问题

• RQ1 Can unsupervised environment design automatically generate a curriculum of solvable, progressively harder environments tailored to a learner's progress?
• RQ2 Does the PAIRED approach produce better zero-shot transfer to novel, unseen environments compared to domain randomization or minimax adversarial training?
• RQ3 What are the theoretical guarantees of minimax regret in this UED setting, and how do they relate to Nash equilibria in the proposed training game?
• RQ4 How does PAIRED impact emergent complexity and policy performance in both discrete navigation and continuous control tasks?

主要发现

• PAIRED generates a curriculum of increasing environmental difficulty that tracks the agent’s learning progress.
• Agents trained with PAIRED show higher zero-shot transfer performance to challenging, novel environments than DR or minimax baselines.
• PAIRED produces more emergent, complex behaviors as measured by increasingly difficult, solvable mazes.
• In continuous control, PAIRED adversaries keep tasks solvable and yield more robust transfer under unseen environment parameters.
• Theoretical results link minimax regret to successful policies when such success is possible and connect the approach to Nash equilibrium concepts.