[Paper Review] Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning

Guozheng Ma, Linrui Zhang+7 more|arXiv (Cornell University)|May 25, 2023

Advanced Fluorescence Microscopy Techniques10 citations

TL;DR

The paper analyzes data augmentation attributes for visual RL and introduces Rand PR and CycAug, achieving superior sample efficiency on DM Control and CARLA without changing RL algorithms.

ABSTRACT

Data augmentation (DA) is a crucial technique for enhancing the sample efficiency of visual reinforcement learning (RL) algorithms. Notably, employing simple observation transformations alone can yield outstanding performance without extra auxiliary representation tasks or pre-trained encoders. However, it remains unclear which attributes of DA account for its effectiveness in achieving sample-efficient visual RL. To investigate this issue and further explore the potential of DA, this work conducts comprehensive experiments to assess the impact of DA's attributes on its efficacy and provides the following insights and improvements: (1) For individual DA operations, we reveal that both ample spatial diversity and slight hardness are indispensable. Building on this finding, we introduce Random PadResize (Rand PR), a new DA operation that offers abundant spatial diversity with minimal hardness. (2) For multi-type DA fusion schemes, the increased DA hardness and unstable data distribution result in the current fusion schemes being unable to achieve higher sample efficiency than their corresponding individual operations. Taking the non-stationary nature of RL into account, we propose a RL-tailored multi-type DA fusion scheme called Cycling Augmentation (CycAug), which performs periodic cycles of different DA operations to increase type diversity while maintaining data distribution consistency. Extensive evaluations on the DeepMind Control suite and CARLA driving simulator demonstrate that our methods achieve superior sample efficiency compared with the prior state-of-the-art methods.

Motivation & Objective

• Investigate which data augmentation attributes (hardness and diversity) drive sample-efficient visual RL.
• Identify limitations of existing multi-type augmentation fusion schemes in RL settings.
• Propose augmentation designs that balance information preservation with spatial diversity.
• Develop RL-friendly fusion strategies that maintain data distribution stability during training.

Proposed method

• Analyze hardness and diversity of DA in visual RL through controlled ablations.
• Propose Random PadResize (Rand PR) to maximize spatial diversity while keeping augmentation hardness low.
• Develop Cycling Augmentation (CycAug), an RL-tailored multi-type DA fusion that cycles between different augmentations to maintain data distribution stability.
• Integrate Rand PR into a DrQ-V2-based pipeline and evaluate on DM Control and CARLA.”

Experimental results

Research questions

• RQ1 What DA attributes (hardness, strength diversity, spatial diversity, type diversity) most influence sample efficiency in visual RL?
• RQ2 Can RL-specific fusion strategies outperform generic multi-type DA fusion schemes?
• RQ3 Do Random PadResize and Cycling Augmentation offer tangible gains in sample efficiency across domains like DM Control and CARLA?
• RQ4 How does controlling data distribution stability affect training performance when using multi-type DA?
• RQ5 What is the impact of DA design on training stability and final performance in challenging RL tasks?

Key findings

• Low augmentation hardness and high spatial diversity are crucial for effective DA in visual RL.
• Unlimited strength diversity can harm performance due to increased hardness.
• Naively applying multi-type DA fusion schemes from CV can reduce RL sample efficiency.
• Rand PR provides diverse yet low-hardness augmentation; CycAug cyclically applying multiple DAs improves stability and sample efficiency.
• CycAug with Rand PR achieves state-of-the-art efficiency on DM Control tasks and outperforms prior SOTA on CARLA, especially in low-data regimes.