[Paper Review] Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning

Denis Yarats, Rob Fergus+2 more|arXiv (Cornell University)|Jul 20, 2021

Reinforcement Learning in RoboticsComputer Science43 references67 citations

TL;DR

DrQ-v2 is a simple model-free RL algorithm for image-based continuous control that uses data augmentation and architectural tweaks to achieve state-of-the-art sample efficiency and enable humanoid locomotion from pixels, with a much improved computational footprint compared to prior methods.

ABSTRACT

We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control. DrQ-v2 builds on DrQ, an off-policy actor-critic approach that uses data augmentation to learn directly from pixels. We introduce several improvements that yield state-of-the-art results on the DeepMind Control Suite. Notably, DrQ-v2 is able to solve complex humanoid locomotion tasks directly from pixel observations, previously unattained by model-free RL. DrQ-v2 is conceptually simple, easy to implement, and provides significantly better computational footprint compared to prior work, with the majority of tasks taking just 8 hours to train on a single GPU. Finally, we publicly release DrQ-v2's implementation to provide RL practitioners with a strong and computationally efficient baseline.

Motivation & Objective

• Motivate and develop a sample-efficient, compute-friendly model-free algorithm for visual continuous control.
• Improve upon DrQ by introducing algorithmic and implementation refinements to solve harder tasks from pixel observations.
• Demonstrate competitive or superior performance to model-based methods while reducing training time.
• Provide an open-source, practical baseline implementation for researchers and practitioners.

Proposed method

• Adopt a DDPG-based actor-critic backbone with two Q-functions for clipped double Q-learning.
• Incorporate n-step returns to accelerate reward propagation.
• Apply data augmentation via random shifts with bilinear interpolation on pixel observations.
• Use a larger replay buffer and tuned hyper-parameters (e.g., smaller batch size, smaller learning rate) for stability and performance.
• Implement a faster image augmentation pipeline using grid_sample to boost throughput.
• Decay exploration noise σ(t) during training to balance exploration and exploitation.

Experimental results

Research questions

• RQ1 Can a model-free algorithm solve complex humanoid locomotion directly from pixel observations?
• RQ2 What combination of algorithmic changes and implementation optimizations yields the best sample efficiency and fastest wall-clock training for visual continuous control?
• RQ3 How does DrQ-v2 compare to model-based methods in terms of sample efficiency and compute requirements?
• RQ4 What are the key factors (e.g., replay buffer size, n-step returns, exploration schedule) that drive performance gains in image-based RL?
• RQ5 Is a simple, scalable baseline for visual RL feasible that democratizes research on this problem?

Key findings

• DrQ-v2 outperforms prior model-free methods in sample efficiency across easy, medium, and hard tasks in the DeepMind Control Suite.
• DrQ-v2 solves humanoid locomotion tasks from pixels, previously unsolved by model-free methods.
• DrQ-v2 trains roughly 3.5x faster in wall-clock time than prior DrQ implementations on the same hardware, with higher environment frame throughput (FPS).
• DrQ-v2 can reach 96 FPS on a single NVIDIA V100 GPU, enabling many tasks to finish in about 2.9–86 hours depending on difficulty.
• Compared to Dreamer-v2 (model-based), DrQ-v2 achieves comparable or better wall-clock efficiency for several tasks, though Dreamer-v2 can outperform on some tasks; overall DrQ-v2 trains faster due to higher throughput.
• Ablations show that switching to a DDPG backbone with 3-step returns, a larger replay buffer, and a decaying exploration schedule substantially boosts performance.