[Paper Review] Time Matters in Using Data Augmentation for Vision-based Deep Reinforcement Learning.

Byungchan Ko, Jungseul Ok|arXiv (Cornell University)|Feb 17, 2021

Reinforcement Learning in Robotics34 references4 citations

TL;DR

This paper investigates the timing of data augmentation in vision-based deep reinforcement learning, demonstrating that delaying augmentation until inference (testing) improves sample and computation efficiency when regularization benefits are only needed at test time. Conversely, augmentations useful during training must be applied throughout training to maximize data efficiency and generalization.

ABSTRACT

Data augmentation technique from computer vision has been widely considered as a regularization method to improve data efficiency and generalization performance in vision-based reinforcement learning. We variate the timing of using augmentation, which is, in turn, critical depending on tasks to be solved in training and testing. According to our experiments on Open AI Procgen Benchmark, if the regularization imposed by augmentation is helpful only in testing, it is better to procrastinate the augmentation after training than to use it during training in terms of sample and computation complexity. We note that some of such augmentations can disturb the training process. Conversely, an augmentation providing regularization useful in training needs to be used during the whole training period to fully utilize its benefit in terms of not only generalization but also data efficiency. These phenomena suggest a useful timing control of data augmentation in reinforcement learning.

Motivation & Objective

• To investigate how the timing of data augmentation impacts training and inference in vision-based deep reinforcement learning.
• To identify whether applying augmentation during training or only at inference yields better sample and computation efficiency.
• To determine under what conditions data augmentation improves generalization and data efficiency.
• To provide practical guidelines for when and how to apply data augmentation in RL based on task-specific regularization needs.

Proposed method

• The authors systematically vary the timing of data augmentation—applying it only during inference, only during training, or throughout both phases.
• Experiments are conducted on the OpenAI Procgen Benchmark to evaluate performance across diverse vision-based RL environments.
• The study compares training dynamics and test performance when augmentation is applied at different stages, measuring sample efficiency and computation cost.
• Regularization effects are analyzed based on whether they are beneficial during training or only at inference.
• The method evaluates both standard augmentation techniques and their impact on policy generalization and learning stability.
• Ablation studies isolate the contribution of augmentation timing, distinguishing between its roles in generalization and data efficiency.

Experimental results

Research questions

• RQ1 When is data augmentation most effective in vision-based deep reinforcement learning—during training or inference?
• RQ2 How does the timing of data augmentation affect sample efficiency and computational complexity?
• RQ3 Under what conditions does augmentation improve generalization versus data efficiency?
• RQ4 Can delaying augmentation until inference reduce training overhead without sacrificing performance?
• RQ5 What determines whether augmentation should be applied throughout training or only at test time?

Key findings

• Applying data augmentation only during inference is more sample- and computation-efficient when regularization is only needed at test time.
• Using augmentation during training is essential for maximizing data efficiency and generalization when the regularization benefits are present during learning.
• Some augmentations can disrupt the training process if applied too early, suggesting timing is critical for stable learning.
• Delaying augmentation until inference reduces training complexity without degrading test performance in cases where augmentation is not needed during training.
• The effectiveness of data augmentation in RL depends on whether its regularization effect is useful during training or only at inference.
• The study provides empirical evidence that timing control of data augmentation is a key factor in balancing efficiency and performance.