[Paper Review] EZClone: Improving DNN Model Extraction Attack via Shape Distillation from GPU Execution Profiles

Jonah O’Brien Weiss, Tiago M. Alves+1 more|arXiv (Cornell University)|Apr 6, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

EZClone is a novel, lightweight DNN model extraction attack that leverages aggregate GPU execution profiles to predict the architecture of a victim deep neural network with 100% accuracy using only three empirically identified GPU kernel features. Unlike prior methods, it does not rely on time-series profiling or strong adversary capabilities, making it highly effective and practical for real-world side-channel threats.

ABSTRACT

Deep Neural Networks (DNNs) have become ubiquitous due to their performance on prediction and classification problems. However, they face a variety of threats as their usage spreads. Model extraction attacks, which steal DNNs, endanger intellectual property, data privacy, and security. Previous research has shown that system-level side-channels can be used to leak the architecture of a victim DNN, exacerbating these risks. We propose two DNN architecture extraction techniques catering to various threat models. The first technique uses a malicious, dynamically linked version of PyTorch to expose a victim DNN architecture through the PyTorch profiler. The second, called EZClone, exploits aggregate (rather than time-series) GPU profiles as a side-channel to predict DNN architecture, employing a simple approach and assuming little adversary capability as compared to previous work. We investigate the effectiveness of EZClone when minimizing the complexity of the attack, when applied to pruned models, and when applied across GPUs. We find that EZClone correctly predicts DNN architectures for the entire set of PyTorch vision architectures with 100% accuracy. No other work has shown this degree of architecture prediction accuracy with the same adversarial constraints or using aggregate side-channel information. Prior work has shown that, once a DNN has been successfully cloned, further attacks such as model evasion or model inversion can be accelerated significantly.

Motivation & Objective

• To address the growing threat of DNN model extraction attacks that compromise intellectual property and data privacy.
• To develop a practical, low-overhead method for extracting DNN architectures using system-level side-channels from GPU execution profiles.
• To overcome limitations of prior methods that rely on time-series data, large candidate sets, or strong adversary assumptions.
• To demonstrate high accuracy in architecture prediction across diverse models, including pruned and cross-GPU deployments.
• To expose a new attack surface via a PyTorch parser that enables architecture leakage through a single code change in a malicious library.

Proposed method

• EZClone extracts aggregate GPU kernel feature values from PyTorch profiler outputs, focusing on non-memory-related kernels to reduce noise.
• It applies Recursive Feature Elimination on a Random Forest model to identify the top three most discriminative GPU kernel features for architecture prediction.
• A Bayesian architecture prediction model is trained on these features to classify DNN architectures with high accuracy.
• A custom PyTorch parser is developed to map high-level layer types and hyperparameters to GPU kernel calls, enabling direct architecture reconstruction from profiles.
• The method is evaluated across 36 PyTorch vision architectures, including pruned models and different GPU platforms, using only one profile per architecture for training.
• The attack is designed to be extensible: new architectures can be added with minimal effort, requiring only a single profile and under one minute of processing.

Figure 1: GPU kernel feature values for all 36 PyTorch vision architectures by architecture family. Values are averaged over 50 profiles and normalized. The features are ranked from the set of non-memory related GPU kernels using Recursive Feature Elimination on a Random Forest model, which achieves

Experimental results

Research questions

• RQ1 Can aggregate GPU execution profiles alone enable accurate DNN architecture prediction without time-series data or strong adversary capabilities?
• RQ2 How does EZClone perform across diverse DNN architectures, including pruned models and different GPU platforms?
• RQ3 Can a minimal set of GPU kernel features (as few as three) achieve 100% architecture prediction accuracy?
• RQ4 To what extent can EZClone be extended to new models with minimal retraining effort?
• RQ5 What are the implications of using a PyTorch parser to enable architecture leakage via a malicious library with only one code change?

Key findings

• EZClone achieves 100% accuracy in predicting all 36 PyTorch vision architectures present in Torchvision v0.10.0 using only three GPU kernel features.
• The method maintains 100% accuracy when applied to pruned models, demonstrating robustness to architectural modifications.
• EZClone achieves 100% cross-GPU accuracy, indicating strong generalization across different GPU platforms.
• The Bayesian architecture prediction model reaches over 72% cross-GPU accuracy even when the attack is weakened to a smaller candidate set.
• The PyTorch parser successfully reconstructs DNN architectures from profiler outputs, revealing a new attack surface via a malicious library with minimal code changes.
• Training a new architecture prediction model for a previously unseen DNN requires only one profile and takes less than one minute to complete.

Figure 2: EZClone attack framework.