[Paper Review] Off-Policy Adversarial Inverse Reinforcement Learning

Samin Yeasar Arnob|arXiv (Cornell University)|May 3, 2020

Reinforcement Learning in Robotics21 references4 citations

TL;DR

This paper proposes Off-Policy Adversarial Inverse Reinforcement Learning (Off-Policy-AIRL), which improves imitation performance and sample efficiency in continuous control tasks by combining Soft Actor-Critic (SAC) with a discriminator-based adversarial framework. The method achieves superior imitation performance over state-of-the-art AIL algorithms and enables effective transfer learning using the learned reward function even without expert demonstrations.

ABSTRACT

Adversarial Imitation Learning (AIL) is a class of algorithms in Reinforcement learning (RL), which tries to imitate an expert without taking any reward from the environment and does not provide expert behavior directly to the policy training. Rather, an agent learns a policy distribution that minimizes the difference from expert behavior in an adversarial setting. Adversarial Inverse Reinforcement Learning (AIRL) leverages the idea of AIL, integrates a reward function approximation along with learning the policy, and shows the utility of IRL in the transfer learning setting. But the reward function approximator that enables transfer learning does not perform well in imitation tasks. We propose an Off-Policy Adversarial Inverse Reinforcement Learning (Off-policy-AIRL) algorithm which is sample efficient as well as gives good imitation performance compared to the state-of-the-art AIL algorithm in the continuous control tasks. For the same reward function approximator, we show the utility of learning our algorithm over AIL by using the learned reward function to retrain the policy over a task under significant variation where expert demonstrations are absent.

Motivation & Objective

• Address the poor imitation performance of existing Adversarial Inverse Reinforcement Learning (AIRL) algorithms despite strong transfer learning capabilities.
• Improve sample efficiency and imitation performance in continuous control tasks by leveraging off-policy deep reinforcement learning algorithms.
• Demonstrate the utility of the learned reward function for policy retraining in transfer learning scenarios where expert demonstrations are unavailable.
• Explore the impact of policy architecture and reward function design on imitation and transfer learning performance.
• Investigate the role of multiplicative compositional policies (MCP) in enhancing flexibility and performance during transfer learning.

Proposed method

• Adopt Soft Actor-Critic (SAC) as the generator (policy) to enable off-policy training and improve sample efficiency.
• Integrate a discriminator network that distinguishes expert trajectories from generated trajectories, forming an adversarial training objective.
• Learn a state-dependent reward function without entropy regularization, which empirically performs best in the proposed framework.
• Use the same reward function approximator for both imitation and transfer learning, ensuring consistency across tasks.
• Apply Multiplicative Compositional Policies (MCP) to allow flexible policy retraining by only updating the gating function during transfer.
• Implement random state initialization during transfer learning to improve exploration and reduce performance variance.

Experimental results

Research questions

• RQ1 Can off-policy deep reinforcement learning methods like SAC improve imitation performance in adversarial inverse reinforcement learning compared to on-policy baselines?
• RQ2 Does the same reward function approximator that performs well in imitation also enable effective transfer learning when expert demonstrations are absent?
• RQ3 Can multiplicative compositional policies (MCP) enhance policy flexibility and performance during transfer learning without full retraining?
• RQ4 How does random state initialization during transfer learning affect performance and variance in off-policy-AIRL?
• RQ5 Why does the method fail when agent dynamics are altered in transfer learning, and what structural properties of agents may influence this failure?

Key findings

• Off-Policy-AIRL with SAC as the generator achieves superior imitation performance compared to state-of-the-art AIL algorithms on MuJoCo continuous control environments such as HalfCheetah-v2 and Ant-v2.
• The use of a simple state-dependent reward network without entropy regularization yields the best performance among tested reward function configurations.
• The learned reward function enables successful policy retraining in transfer learning tasks even without expert demonstrations, demonstrating utility beyond imitation.
• SAC-MCP improves imitation performance and reduces performance variance in transfer learning compared to standard SAC, especially when only the gating function is fine-tuned.
• Random state initialization during transfer learning enhances exploration and reduces performance variance, leading to more stable and higher-performing policies.
• The method fails to generalize when agent dynamics are altered (e.g., paralyzed legs in Ant), suggesting that structural properties of the agent significantly impact transferability.