[Paper Review] CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Aravind Srinivas, Michael Laskin+1 more|arXiv (Cornell University)|Apr 8, 2020

Reinforcement Learning in RoboticsComputer Science394 citations

TL;DR

CURL couples contrastive self-supervised representation learning with off-policy reinforcement learning to improve data efficiency from pixel inputs, achieving substantial gains on DMControl and Atari benchmarks compared to prior pixel-based methods.

ABSTRACT

We present CURL: Contrastive Unsupervised Representations for Reinforcement Learning. CURL extracts high-level features from raw pixels using contrastive learning and performs off-policy control on top of the extracted features. CURL outperforms prior pixel-based methods, both model-based and model-free, on complex tasks in the DeepMind Control Suite and Atari Games showing 1.9x and 1.2x performance gains at the 100K environment and interaction steps benchmarks respectively. On the DeepMind Control Suite, CURL is the first image-based algorithm to nearly match the sample-efficiency of methods that use state-based features. Our code is open-sourced and available at https://github.com/MishaLaskin/curl.

Motivation & Objective

• Motivate the need for data-efficient learning from high-dimensional pixel inputs in RL.
• Propose a simple, plug-in contrastive learning framework to learn representations jointly with RL without heavy architectural changes.
• Demonstrate data-efficiency and performance gains over prior pixel-based methods on DMControl and Atari tasks.
• Show that contrastive objectives can match state-based methods in sample efficiency for image-based control.

Proposed method

• Use instance discrimination with augmented frame stacks to learn representations via a contrastive loss (InfoNCE).
• Employ a momentum-averaged target encoder (MoCo-style) with a bilinear similarity (q^T W k) for the contrastive score.
• Train the RL policy/value function on top of the query encoder while the target encoder provides keys without gradient flow.
• Apply simple data augmentations (random crops) across frame stacks to generate query-key pairs while preserving temporal structure.
• Couple CURL with an existing RL algorithm (SAC for DMControl; data-efficient Rainbow for Atari) with minimal extra hyperparameters.
• Demonstrate that contrastive learning acts as an auxiliary task improving sample-efficiency without requiring world-models or reconstruction losses.

Experimental results

Research questions

• RQ1 Does a simple contrastive self-supervised objective improve data-efficiency for pixel-based RL tasks?
• RQ2 Can CURL achieve competitive or superior sample efficiency and performance compared to state-of-the-art pixel-based methods on DMControl and Atari?
• RQ3 Is it possible to integrate contrastive learning with off-policy RL in a simple, reproducible pipeline without heavy architectural changes?
• RQ4 How does momentum-encoded keys and a bilinear similarity affect representation quality and downstream control performance?

Key findings

• CURL achieves 1.9x median higher performance than the prior state-of-the-art on DMControl at 100k steps.
• CURL achieves 1.2x median higher performance than prior methods on Atari at 100k steps.
• On DMControl, CURL is the first image-based method to nearly match sample-efficiency of state-based SAC in many environments.
• Across pixel-based tasks, CURL shows substantial data-efficiency gains over both model-based and model-free baselines.
• The authors emphasize the simplicity and reproducibility of CURL, avoiding complex architectural changes or heavy additional hyperparameters.
• CURL demonstrates strong empirical results across 16 DMControl tasks and 26 Atari games with pixel inputs.