[Paper Review] Skew-Fit: State-Covering Self-Supervised Reinforcement Learning

Vitchyr H. Pong, Murtaza Dalal+4 more|arXiv (Cornell University)|Mar 8, 2019

Reinforcement Learning in Robotics39 references66 citations

TL;DR

Skew-Fit proposes a principled objective for unsupervised RL that maximizes state coverage by learning a maximum-entropy goal distribution and pairs it with goal-directed exploration to yield reusable policies for reaching user-specified goals.

ABSTRACT

Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manually-designed reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing goal reaching performance together with the entropy of the goal distribution, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. We prove that, under regularity conditions, Skew-Fit converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that combining Skew-Fit for learning goal distributions with existing goal-reaching methods outperforms a variety of prior methods on open-sourced visual goal-reaching tasks. Moreover, we demonstrate that Skew-Fit enables a real-world robot to learn to open a door, entirely from scratch, from pixels, and without any manually-designed reward function.

Motivation & Objective

• Motivate the need for broad skill repertoires without manual reward design.
• Define an exploration objective that maximizes state coverage while enabling test-time goal-directed use.
• Propose Skew-Fit to learn a maximum-entropy goal distribution over valid states.
• Demonstrate convergence guarantees to a uniform goal distribution under regularity assumptions.
• Show empirical gains on vision-based tasks and a real-world door-opening robot.

Proposed method

• Frame exploration as maximizing the mutual information I(S;G) between states S and goals G.
• Decompose into minimizing H(G|S) via goal-conditioned RL and maximizing H(G) via diverse goal sampling.
• Introduce Skew-Fit: iteratively skew samples toward rare states to train a generative model q_phi^G that approaches a uniform distribution over S.
• Approximate p^S_phi(S) with q_phi^G(S) to compute importance-sampling based weights w_t,alpha(S)=q_phi^G(S)^alpha with alpha<0.
• Use sampling importance resampling (SIR) to reduce variance when updating q_phi^G.
• Concurrent training of a goal-conditioned policy (e.g., via RIG with a beta-VAE) to minimize H(G|S) and relabel goals.
• Reuse the beta-VAE as the generative model for q_phi^G and leverage learned representations for image-based goals.

Experimental results

Research questions

• RQ1 Does Skew-Fit reliably increase the entropy of the goal distribution over iterations?
• RQ2 Does coupling Skew-Fit with a goal-conditioned policy improve exploration for vision-based RL tasks?
• RQ3 How does Skew-Fit compare to prior goal-sampling strategies on vision-based continuous control tasks?
• RQ4 Can Skew-Fit scale to real-world vision-based robotic manipulation without task rewards?
• RQ5 Under what conditions does Skew-Fit converge to a uniform distribution over valid states?

Key findings

• Skew-Fit steadily increases the state-space entropy of the goal distribution, approaching near-uniform coverage in a 2D navigation example.
• In ant navigation and labyrinth tasks, Skew-Fit significantly outperforms prior goal-sampling methods in exploration efficiency.
• On vision-based continuous control tasks (Visual Door, Visual Pusher, Visual Pickup) and a real-world Visual Door task, Skew-Fit outperforms baselines and enables door opening from pixels without task rewards.
• Skew-Fit enables learning from scratch with no manually-designed rewards, achieving a real-world door-opening capability in about five hours of training.
• The method reuses a beta-VAE-based representation (RIG) to train both goal-conditioned policies and the goal distribution, leveraging relabeling and latent space planning.