[Paper Review] Sample-efficient Adversarial Imitation Learning from Observation

Faraz Torabi, Sean Geiger+2 more|arXiv (Cornell University)|Jun 18, 2019

Adversarial Robustness in Machine Learning25 references5 citations

TL;DR

This paper proposes LQR+GAIfO, a sample-efficient imitation learning algorithm that combines trajectory-centric reinforcement learning (LQR) with generative adversarial imitation from observation (GAIfO) to improve learning efficiency. By using LQR to generate high-quality, low-level trajectories that guide the adversarial policy training, the method achieves faster convergence and superior sample efficiency compared to GAIfO alone, enabling successful imitation on physical robotic systems with fewer demonstrations and training iterations.

ABSTRACT

Imitation from observation is the framework of learning tasks by observing demonstrated state-only trajectories. Recently, adversarial approaches have achieved significant performance improvements over other methods for imitating complex behaviors. However, these adversarial imitation algorithms often require many demonstration examples and learning iterations to produce a policy that is successful at imitating a demonstrator's behavior. This high sample complexity often prohibits these algorithms from being deployed on physical robots. In this paper, we propose an algorithm that addresses the sample inefficiency problem by utilizing ideas from trajectory centric reinforcement learning algorithms. We test our algorithm and conduct experiments using an imitation task on a physical robot arm and its simulated version in Gazebo and will show the improvement in learning rate and efficiency.

Motivation & Objective

• To address the high sample complexity of adversarial imitation learning (GAIfO) in real-world robotic applications.
• To improve sample efficiency in imitation from observation (IfO) where only state trajectories are available, without action information.
• To enable effective imitation learning on physical robots by reducing the number of required demonstrations and training iterations.
• To combine the sample efficiency of trajectory-centric LQR with the policy capacity of deep neural networks in GAIfO.
• To validate the method on both simulated and physical 6-DOF robot arms, demonstrating transferability from simulation to reality.

Proposed method

• Integrates Linear Quadratic Regulators (LQR) to generate high-quality, low-level trajectory controllers that guide the policy search.
• Uses the LQR-generated trajectories as a prior to initialize and guide the generator network in the GAIfO framework.
• Employs a generative adversarial network (GAN) setup where a discriminator distinguishes expert trajectories from imitator trajectories.
• Trains the generator (imitator policy) adversarially using the discriminator’s feedback, while leveraging LQR for sample-efficient exploration.
• Applies importance-weighted sampling from LQR trajectories to improve policy update efficiency and generalization.
• Extends the framework to accept visual observations by adding convolutional layers to the discriminator, enabling imitation from video data.

Experimental results

Research questions

• RQ1 Can combining LQR with GAIfO significantly reduce the number of samples required for successful imitation from observation?
• RQ2 How does the performance of LQR+GAIfO compare to GAIfO alone in terms of sample efficiency and final imitation accuracy?
• RQ3 To what extent can policies trained in simulation transfer successfully to a physical robot arm?
• RQ4 Does the integration of LQR improve generalization to new target points not seen during training?
• RQ5 Can the method be extended to work with visual observations alone, enabling imitation from video demonstrations?

Key findings

• LQR+GAIfO achieves successful imitation with significantly fewer training iterations and demonstration samples than GAIfO alone, demonstrating improved sample efficiency.
• The method shows faster learning convergence, with performance surpassing GAIfO in early training stages, particularly within the first 60 iterations.
• Despite eventual performance degradation in LQR+GAIfO around iteration 60, likely due to mismatched training rates between the discriminator and controller, the method still outperforms GAIfO in early sample-limited regimes.
• The policy trained in simulation successfully generalizes to the physical robot arm, with real-world performance exceeding simulator results, possibly due to increased exploration from physical noise.
• The integration of LQR enables better generalization to new target points, especially as more demonstration trajectories are provided, due to the discriminator learning a general cost function.
• Future extension to visual observation via convolutional discriminators is a promising direction, as it would allow imitation from video data without requiring action annotations.