[Paper Review] Hermes Attack: Steal DNN Models with Lossless Inference Accuracy

Yuankun Zhu, Yueqiang Cheng+2 more|arXiv (Cornell University)|Jun 23, 2020

Adversarial Robustness in Machine LearningComputer Science43 references36 citations

TL;DR

Hermes Attack is a two-phase black-box method that fully reconstructs a victim DNN model by eavesdropping unencrypted PCIe traffic, achieving lossless inference accuracy.

ABSTRACT

Deep Neural Networks (DNNs) models become one of the most valuable enterprise assets due to their critical roles in all aspects of applications. With the trend of privatization deployment of DNN models, the data leakage of the DNN models is becoming increasingly serious and widespread. All existing model-extraction attacks can only leak parts of targeted DNN models with low accuracy or high overhead. In this paper, we first identify a new attack surface -- unencrypted PCIe traffic, to leak DNN models. Based on this new attack surface, we propose a novel model-extraction attack, namely Hermes Attack, which is the first attack to fully steal the whole victim DNN model. The stolen DNN models have the same hyper-parameters, parameters, and semantically identical architecture as the original ones. It is challenging due to the closed-source CUDA runtime, driver, and GPU internals, as well as the undocumented data structures and the loss of some critical semantics in the PCIe traffic. Additionally, there are millions of PCIe packets with numerous noises and chaos orders. Our Hermes Attack addresses these issues by huge reverse engineering efforts and reliable semantic reconstruction, as well as skillful packet selection and order correction. We implement a prototype of the Hermes Attack, and evaluate two sequential DNN models (i.e., MINIST and VGG) and one consequential DNN model (i.e., ResNet) on three NVIDIA GPU platforms, i.e., NVIDIA Geforce GT 730, NVIDIA Geforce GTX 1080 Ti, and NVIDIA Geforce RTX 2080 Ti. The evaluation results indicate that our scheme is able to efficiently and completely reconstruct ALL of them with making inferences on any one image. Evaluated with Cifar10 test dataset that contains 10,000 images, the experiment results show that the stolen models have the same inference accuracy as the original ones (i.e., lossless inference accuracy).

Motivation & Objective

• Identify a new attack surface for DNN model leakage via unencrypted PCIe traffic in privatization deployments.
• Propose Hermes Attack to fully reconstruct the victim DNN model including architecture, hyper-parameters, and parameters.
• Demonstrate practicality by implementing and evaluating on real GPU platforms with multiple model types.
• Show that reconstructed models maintain the original inference accuracy on standard datasets.

Proposed method

• Offline phase builds a knowledge database by reverse engineering GPU command headers, kernel-to-layer mappings, and kernel-to-hyper-parameter offsets.
• Online phase uses intercepted PCIe traffic to reconstruct the full DNN model by filtering commands, locating kernel binaries, and assembling a data-flow graph.
• Kernel binaries are located by correlating K commands (kernel launches) with D commands (data moves) during profiling.
• Semantic reconstruction links GPU kernels to layer types and extracts hyper-parameters from kernel arguments.
• Model reconstruction builds a data-flow graph from command edges and substitutes kernels with corresponding DNN layers using the knowledge base.
• Parameters are recovered from D and K/D2D commands, including handling of large and asynchronous data transfers.

Experimental results

Research questions

• RQ1 Can unencrypted PCIe traffic be exploited to completely reconstruct a victim DNN model (architecture, hyper-parameters, and parameters) in privatization deployments?
• RQ2 Is it possible to achieve lossless inference accuracy on reconstructed models across sequential and non-sequential architectures on real GPUs?
• RQ3 What practical steps and reverse-engineering techniques are required to map GPU commands to DNN components and parameters?
• RQ4 How quickly can a complete model be stolen from a single inference instance?

Key findings

• The Hermes Attack can completely reconstruct the victim model, including architecture, hyper-parameters, and parameters.
• The stolen models achieve lossless inference accuracy, matching the original model on 10,000 CIFAR-10/analogous test images.
• The attack was demonstrated on three NVIDIA GPU platforms (GT 730, GTX 1080 Ti, RTX 2080 Ti) using MNIST, VGG, and ResNet models.
• Reconstruction from a single image inference completes within 5–17 minutes.
• Only about 1–2% of captured PCIe packets are useful for the reconstruction, with the rest treated as noise.
• A two-phase process (offline knowledge base construction and online traffic-driven reconstruction) enables the full model theft.