[Paper Review] Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning

Nicholas E. Corrado, Y. Qu+3 more|arXiv (Cornell University)|Oct 27, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

GuDA is a human-guided data augmentation framework that generates expert-quality demonstration data by applying user-defined rules to select trajectory segments that make progress toward task completion. It significantly outperforms random data augmentation in offline reinforcement learning and imitation learning, enabling effective policy learning from just a few suboptimal demonstrations, including in a physical robot soccer task where it surpassed the expert policy.

ABSTRACT

In offline reinforcement learning (RL), an RL agent learns to solve a task using only a fixed dataset of previously collected data. While offline RL has been successful in learning real-world robot control policies, it typically requires large amounts of expert-quality data to learn effective policies that generalize to out-of-distribution states. Unfortunately, such data is often difficult and expensive to acquire in real-world tasks. Several recent works have leveraged data augmentation (DA) to inexpensively generate additional data, but most DA works apply augmentations in a random fashion and ultimately produce highly suboptimal augmented experience. In this work, we propose Guided Data Augmentation (GuDA), a human-guided DA framework that generates expert-quality augmented data. The key insight behind GuDA is that while it may be difficult to demonstrate the sequence of actions required to produce expert data, a user can often easily characterize when an augmented trajectory segment represents progress toward task completion. Thus, a user can restrict the space of possible augmentations to automatically reject suboptimal augmented data. To extract a policy from GuDA, we use off-the-shelf offline reinforcement learning and behavior cloning algorithms. We evaluate GuDA on a physical robot soccer task as well as simulated D4RL navigation tasks, a simulated autonomous driving task, and a simulated soccer task. Empirically, GuDA enables learning given a small initial dataset of potentially suboptimal experience and outperforms a random DA strategy as well as a model-based DA strategy.

Motivation & Objective

• To address the challenge of acquiring high-quality expert demonstrations, which are costly and difficult to collect in real-world tasks.
• To improve the performance of offline reinforcement learning and imitation learning by generating diverse, expert-like demonstration data without additional real-world interaction.
• To reduce reliance on optimal expert demonstrations by enabling practitioners to guide data augmentation using simple, intuitive rules about task progress.
• To evaluate the effectiveness of guided data augmentation in both simulated and real-world environments, including complex physical robotics tasks.

Proposed method

• GuDA uses a set of user-defined rules to filter and generate augmented trajectory segments that represent meaningful progress toward task completion, rather than sampling randomly.
• The framework applies geometric transformations—such as translation, rotation, and reflection—to existing demonstration trajectories to generate new, diverse trajectories.
• Users specify rules based on task-relevant behaviors (e.g., approaching a goal, orienting correctly), ensuring generated data approximates expert behavior.
• The method leverages off-the-shelf offline RL and behavior cloning algorithms to train policies from the augmented dataset.
• GuDA is applied to tasks including simulated navigation, autonomous driving, and a physical robot soccer environment with a single suboptimal demonstration.
• The approach is evaluated using success rates and time-to-completion metrics, comparing GuDA against random data augmentation and no augmentation.

Experimental results

Research questions

• RQ1 Can user-guided data augmentation generate expert-quality demonstration data without requiring real-world interaction?
• RQ2 How does guided data augmentation compare to random data augmentation in terms of policy performance in offline reinforcement learning and imitation learning?
• RQ3 Can GuDA enable effective policy learning from only a few, even suboptimal, expert demonstrations?
• RQ4 Does GuDA outperform the original demonstrator in real-world physical tasks when trained on augmented data?

Key findings

• In the physical robot soccer task with a single suboptimal demonstration, GuDA achieved a success rate of 7/10 under hard initialization, while Random DA and No DA achieved 0/10.
• With easy initialization, GuDA achieved a success rate of 8/10, significantly outperforming Random DA (4/10) and No DA (4/10), and matched the expert’s 9/10 success rate.
• GuDA-trained policies scored faster than the expert policy in the easy initialization setting, indicating improved sample efficiency and policy quality.
• In the hard initialization, only the GuDA policy consistently succeeded, while the expert policy failed in nearly all trials, demonstrating that GuDA can surpass the demonstrator.
• GuDA-generated data led to policies that generalized better and achieved lower time-to-score compared to random data augmentation, as shown by IQM metrics and confidence intervals.
• The results confirm that GuDA enables effective learning from minimal, suboptimal demonstrations by generating high-quality, task-progress-focused augmented data.