[Paper Review] COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration

Nicholas Watters, Löıc Matthey+3 more|arXiv (Cornell University)|May 22, 2019

Reinforcement Learning in Robotics36 references67 citations

TL;DR

COBRA combines unsupervised object-centric representation learning, curiosity-driven exploration, and model-based RL to achieve data-efficient and robust control in a continuous action environment. It uses an unsupervised exploration phase to learn object representations and dynamics, then solves tasks via a model-based reward predictor with 1-step planning.

ABSTRACT

Data efficiency and robustness to task-irrelevant perturbations are long-standing challenges for deep reinforcement learning algorithms. Here we introduce a modular approach to addressing these challenges in a continuous control environment, without using hand-crafted or supervised information. Our Curious Object-Based seaRch Agent (COBRA) uses task-free intrinsically motivated exploration and unsupervised learning to build object-based models of its environment and action space. Subsequently, it can learn a variety of tasks through model-based search in very few steps and excel on structured hold-out tests of policy robustness.

Motivation & Objective

• Motivate data efficiency and robustness in deep reinforcement learning without hand-crafted supervision.
• Develop object-centric representations discovered from raw pixels.
• Integrate curiosity-driven exploration with a transition model trained in an unsupervised phase.
• Enable rapid task solving via model-based planning using prelearned world models.

Proposed method

• Use MONet to obtain a slot-based, object-centric scene representation from pixels.
• Train an action-conditioned, slot-wise transition model to predict next-step object representations.
• Leverage an adversarial exploration policy that moves objects to maximize transition-model prediction error.
• Parameterize a distribution over the continuous 4D action space to sample informative actions for exploration.
• During the task phase, freeze vision/transition/exploration modules and train a reward predictor; perform 1-step model-predictive control by evaluating sampled actions via the predicted next state and reward.

Experimental results

Research questions

• RQ1 Can unsupervised object discovery yield representations that improve data efficiency for downstream tasks?
• RQ2 Does curiosity-driven, adversarial exploration improve dynamics learning and robustness in object-centered RL?
• RQ3 How effective is a task-agnostic, model-based search using precomputed world models for rapid task solving in continuous control?
• RQ4 What is the impact of joint object-centric representations on policy robustness to task-irrelevant perturbations?

Key findings

• COBRA achieves high data efficiency, solving tasks with very few on-task environment steps compared to baselines.
• The agent demonstrates robustness to task-irrelevant perturbations across held-out tests.
• Unsupervised exploration coupled with an object-centric transition model enables rapid policy learning via model-based search.
• Learning a reward predictor on top of frozen world models yields effective one-step MPC policies.
• The approach amortizes pretraining across multiple tasks, reducing the need for task-specific policy training.