[Paper Review] A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning

Guozheng Ma, Zhen Wang+4 more|arXiv (Cornell University)|Oct 10, 2022

Tactile and Sensory Interactions17 citations

TL;DR

This survey presents a unified framework for data augmentation (DA) in visual reinforcement learning (RL), systematically categorizing augmentation techniques and their integration with RL. It demonstrates that DA significantly improves sample efficiency and zero-shot generalization across benchmarks like DMControl and Procgen, with empirical validation showing consistent performance gains when combined with contrastive learning and auxiliary tasks.

ABSTRACT

Visual reinforcement learning (RL), which makes decisions directly from high-dimensional visual inputs, has demonstrated significant potential in various domains. However, deploying visual RL techniques in the real world remains challenging due to their low sample efficiency and large generalization gaps. To tackle these obstacles, data augmentation (DA) has become a widely used technique in visual RL for acquiring sample-efficient and generalizable policies by diversifying the training data. This survey aims to provide a timely and essential review of DA techniques in visual RL in recognition of the thriving development in this field. In particular, we propose a unified framework for analyzing visual RL and understanding the role of DA in it. We then present a principled taxonomy of the existing augmentation techniques used in visual RL and conduct an in-depth discussion on how to better leverage augmented data in different scenarios. Moreover, we report a systematic empirical evaluation of DA-based techniques in visual RL and conclude by highlighting the directions for future research. As the first comprehensive survey of DA in visual RL, this work is expected to offer valuable guidance to this emerging field.

Motivation & Objective

• To address low sample efficiency and large generalization gaps in visual RL by systematically reviewing data augmentation (DA) techniques.
• To propose a unified High-Dimensional Contextual MDP (HCMDP) framework to formalize visual RL and analyze the role of DA.
• To establish a principled taxonomy of DA methods in visual RL, distinguishing between observation, transition, trajectory, and automatic augmentation.
• To investigate how augmented data can be effectively leveraged via implicit/explicit regularization and representation learning.
• To identify open challenges and future directions, including semantic-level augmentation and theoretical foundations for DA in RL.

Proposed method

• Proposes High-Dimensional Contextual MDP (HCMDP) as a formal framework to model visual RL and analyze sample efficiency and generalization.
• Classifies DA techniques into observation-level (geometric, photometric, noise, image mixing, random erasing) and DNN-based (feature space, adversarial, GAN-based) transformations.
• Introduces transition and trajectory-level augmentation to preserve temporal and environmental structure in augmented data.
• Proposes automatic augmentation strategies using search or learnable policies to optimize augmentation policies.
• Introduces context-aware augmentation that adapts to environment-specific priors, such as background variations or object properties.
• Employs multiple learning paradigms to leverage augmented data: implicit regularization, explicit auxiliary tasks (consistency, contrastive learning, future prediction), and task-agnostic representation learning.

Experimental results

Research questions

• RQ1 How can data augmentation be systematically categorized and unified within the visual RL paradigm?
• RQ2 What are the key mechanisms through which data augmentation improves sample efficiency and generalization in visual RL?
• RQ3 How do different augmentation strategies (e.g., geometric vs. GAN-based) compare in performance across standard visual RL benchmarks?
• RQ4 What is the role of auxiliary tasks such as contrastive learning and future prediction in leveraging augmented data?
• RQ5 What are the theoretical and practical limitations of data augmentation in visual RL, and how can they be addressed?

Key findings

• Data augmentation significantly improves sample efficiency on DMControl-100k and DMControl-500k, with consistent gains across multiple algorithms.
• On Procgen, DA-based methods achieve strong zero-shot generalization, particularly under level generalization, due to effective capture of environment priors.
• Contrastive learning with DA leads to more robust and disentangled representations, improving performance by reducing reliance on spurious features.
• Automatic and context-aware augmentation outperform hand-designed strategies in environments with known domain shifts, such as background variations in DMControl-GB.
• Despite empirical success, DA can sometimes increase uncertainty in estimates, highlighting a need for better theoretical grounding in RL settings.
• The integration of DA with pretraining and auxiliary tasks consistently improves performance, suggesting synergistic benefits across representation learning and policy optimization.