[Paper Review] Preventing Imitation Learning with Adversarial Policy Ensembles

Albert Zhan, Stas Tiomkin+1 more|arXiv (Cornell University)|Jan 31, 2020

Reinforcement Learning in Robotics40 references4 citations

TL;DR

This paper introduces Adversarial Policy Ensembles (APE), a novel reinforcement learning framework that trains a collection of near-optimal policies to actively resist imitation learning by an external observer. By modifying policy gradient optimization to simultaneously maximize ensemble performance and minimize the performance of any cloned policy, APE achieves over 5x degradation in cloned policy returns while maintaining near-optimal performance for the original ensemble.

ABSTRACT

Imitation learning can reproduce policies by observing experts, which poses a problem regarding policy privacy. Policies, such as human, or policies on deployed robots, can all be cloned without consent from the owners. How can we protect against external observers cloning our proprietary policies? To answer this question we introduce a new reinforcement learning framework, where we train an ensemble of near-optimal policies, whose demonstrations are guaranteed to be useless for an external observer. We formulate this idea by a constrained optimization problem, where the objective is to improve proprietary policies, and at the same time deteriorate the virtual policy of an eventual external observer. We design a tractable algorithm to solve this new optimization problem by modifying the standard policy gradient algorithm. Our formulation can be interpreted in lenses of confidentiality and adversarial behaviour, which enables a broader perspective of this work. We demonstrate the existence of "non-clonable" ensembles, providing a solution to the above optimization problem, which is calculated by our modified policy gradient algorithm. To our knowledge, this is the first work regarding the protection of policies in Reinforcement Learning.

Motivation & Objective

• To address the critical gap in reinforcement learning privacy by protecting proprietary policies from unauthorized cloning.
• To develop a method that ensures expert demonstrations remain useful for the original policy owner but useless for external observers attempting behavioral cloning.
• To formulate a constrained optimization problem that jointly improves policy performance and degrades the performance of any potential cloned policy.
• To design a tractable algorithm based on policy gradient that enables training of such adversarial policy ensembles in practice.
• To demonstrate that policy ensembles can be engineered to be non-clonable, even under significant data collection efforts by an external observer.

Proposed method

• Formulate the problem as a constrained optimization: maximize expected return of the policy ensemble while minimizing the expected return of a hypothetical cloned policy.
• Introduce a fictitious observer policy that learns from the ensemble's demonstrations, modeling the worst-case scenario for imitation.
• Modify the standard policy gradient algorithm to estimate gradients not only for the ensemble policies but also for the observer's virtual policy.
• Use Monte Carlo sampling to approximate the observer's policy gradient, enabling joint optimization of both the ensemble and the observer's performance.
• Incorporate a context variable in the policy representation to marginalize out expert identity, making it impossible for the observer to infer individual expert behavior.
• Apply the algorithm in discrete MDP environments to train policy ensembles that are robust to behavioral cloning.

Experimental results

Research questions

• RQ1 Can a set of expert policies be designed such that their collective demonstrations are useless for an external observer attempting to clone them via behavioral cloning?
• RQ2 Is it possible to simultaneously optimize for high performance of the original policy ensemble and low performance of any cloned policy derived from it?
• RQ3 How can the policy gradient algorithm be extended to jointly optimize both the ensemble and a virtual observer policy in a differentiable and tractable manner?
• RQ4 What is the quantitative degradation in performance of a cloned policy compared to the original ensemble when using the proposed method?
• RQ5 Can the framework be generalized to continuous state spaces and different definitions of 'uselessness' for cloned policies?

Key findings

• The proposed APE method successfully trains policy ensembles that achieve near-optimal performance while causing a cloned policy to suffer over 5 times greater performance degradation compared to the optimal policy.
• In numerical simulations, the APE method resulted in a performance difference of +0.07 between the ensemble and the cloned policy, indicating that the clone is significantly worse than the original ensemble.
• All baseline policy ensembles showed negligible performance difference between the ensemble and the cloned policy, confirming that APE is uniquely effective in preventing cloning.
• The method remains effective even when the observer collects a large number of demonstrations, demonstrating robustness against data-rich cloning attempts.
• Preliminary results suggest that if the observer uses a strong sequential model like an RNN, the adversarial effect diminishes, indicating a limit to the method's effectiveness under advanced imitation strategies.
• The framework can be extended to use alternative reward functions to define 'uselessness' beyond expected return, enabling customization of adversarial behavior.