[Paper Review] Stealing Deep Reinforcement Learning Models for Fun and Profit

Kangjie Chen, Shangwei Guo+3 more|arXiv (Cornell University)|Jun 9, 2020

Adversarial Robustness in Machine Learning53 references4 citations

TL;DR

This paper presents the first model extraction attack against Deep Reinforcement Learning (DRL) models, enabling an adversary to recover a black-box DRL policy with high fidelity using only action queries. By leveraging imitation learning and a novel algorithm classifier trained on action timing sequences, the attack achieves precise model reconstruction and bypasses defenses like watermarks and query-rate detection.

ABSTRACT

This paper presents the first model extraction attack against Deep Reinforcement Learning (DRL), which enables an external adversary to precisely recover a black-box DRL model only from its interaction with the environment. Model extraction attacks against supervised Deep Learning models have been widely studied. However, those techniques cannot be applied to the reinforcement learning scenario due to DRL models' high complexity, stochasticity and limited observable information. We propose a novel methodology to overcome the above challenges. The key insight of our approach is that the process of DRL model extraction is equivalent to imitation learning, a well-established solution to learn sequential decision-making policies. Based on this observation, our methodology first builds a classifier to reveal the training algorithm family of the targeted black-box DRL model only based on its predicted actions, and then leverages state-of-the-art imitation learning techniques to replicate the model from the identified algorithm family. Experimental results indicate that our methodology can effectively recover the DRL models with high fidelity and accuracy. We also demonstrate two use cases to show that our model extraction attack can (1) significantly improve the success rate of adversarial attacks, and (2) steal DRL models stealthily even they are protected by DNN watermarks. These pose a severe threat to the intellectual property and privacy protection of DRL applications.

Motivation & Objective

• To investigate whether proprietary DRL models can be extracted from black-box access via action-only queries.
• To address the challenges of high complexity, stochasticity, and limited observability in DRL model extraction.
• To develop a method that enables high-fidelity model recovery without access to confidence scores, gradients, or side-channel data.
• To demonstrate the practical impact of such attacks by enhancing adversarial attacks and bypassing digital watermarks.

Proposed method

• Formalize DRL model extraction as an imitation learning problem, where the goal is to mimic the behavior of a black-box DRL policy.
• Design a recurrent neural network (RNN)-based classifier to predict the training algorithm family of the target DRL model using only its action sequences.
• Use the predicted algorithm family as a prior to guide imitation learning, improving fidelity and convergence of the extracted model.
• Apply state-of-the-art imitation learning techniques, such as Generative Adversarial Imitation Learning (GAIL), to train the surrogate policy from expert demonstrations.
• Collect action sequences from the target DRL model in a simulated environment to train the algorithm classifier and generate expert trajectories for imitation.
• Integrate the algorithm classifier and imitation learning pipeline to enable end-to-end model extraction with minimal query overhead.

Experimental results

Research questions

• RQ1 Can a black-box DRL model be extracted using only action outputs, without access to confidence scores, gradients, or side-channel information?
• RQ2 How can the inherent stochasticity and temporal dependencies in DRL policies be leveraged to infer the underlying training algorithm family?
• RQ3 To what extent does prior knowledge of the training algorithm family improve the fidelity and accuracy of model extraction?
• RQ4 Can the extracted model be used to enhance adversarial attacks or bypass existing IP protection mechanisms like digital watermarks?

Key findings

• The proposed method successfully recovers DRL models with high fidelity and accuracy, even when the target model uses complex, deep architectures.
• The algorithm classifier achieves high accuracy in predicting the training algorithm family using only action sequences, enabling effective prior-guided imitation learning.
• The extracted models can significantly improve the success rate of adversarial attacks, demonstrating the practical threat of model extraction.
• The attack successfully bypasses digital watermarking mechanisms, as the extracted models do not retain the original watermarks.
• The attack remains effective under normal query patterns, evading detection mechanisms based on query-rate analysis.