[Paper Review] Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks

Tribhuvanesh Orekondy, Bernt Schiele+1 more|arXiv (Cornell University)|Jun 26, 2019

Adversarial Robustness in Machine LearningComputer Science26 references65 citations

TL;DR

The paper proposes MAD, an active defense that perturbs black-box predictions to poison a stealing attacker’s training objective, substantially reducing stolen-model accuracy with minimal impact on benign accuracy.

ABSTRACT

High-performance Deep Neural Networks (DNNs) are increasingly deployed in many real-world applications e.g., cloud prediction APIs. Recent advances in model functionality stealing attacks via black-box access (i.e., inputs in, predictions out) threaten the business model of such applications, which require a lot of time, money, and effort to develop. Existing defenses take a passive role against stealing attacks, such as by truncating predicted information. We find such passive defenses ineffective against DNN stealing attacks. In this paper, we propose the first defense which actively perturbs predictions targeted at poisoning the training objective of the attacker. We find our defense effective across a wide range of challenging datasets and DNN model stealing attacks, and additionally outperforms existing defenses. Our defense is the first that can withstand highly accurate model stealing attacks for tens of thousands of queries, amplifying the attacker's error rate up to a factor of 85$ imes$ with minimal impact on the utility for benign users.

Motivation & Objective

• Motivate and formalize model functionality stealing via black-box querying and the associated risks to valuable DNN models.
• Propose an active defense that perturbs predictions to poison the attacker’s training objective while preserving defender utility.
• Develop a practical solver to generate perturbations under utility and simplex constraints.
• Demonstrate robustness of the defense across multiple victim models, datasets, and stealing attacks.
• Compare against baseline defenses to establish improvements in non-replicability and efficiency of attack mitigation.

Proposed method

• Formulate defense as maximizing angular deviation between the attacker’s gradient and the defender’s poisoned-gradient, under a utility constraint.
• Model the attacker gradient using a surrogate Jacobian G of log probabilities with respect to the defender’s parameters.
• Perturb posterior predictions y to obtain y~ that maximize the angular deviation while keeping y~ within the probability simplex and within a perturbation budget epsilon.
• Approximate the gradient-based optimization via a two-step solver that searches extreme points of the simplex and forms perturbations by convex combination with the original y.
• Introduce a MAD-argmax variant that additionally preserves the top-1 label by constraining the perturbation to align with the argmax.
• Evaluate using six victim models across several datasets and four stealing attacks, comparing to baseline defenses like reverse-sigmoid, random noise, and DP-SGD.

Experimental results

Research questions

• RQ1 Can an active perturbation of predictions effectively degrade a stolen model’s performance without overly hurting the defender’s utility?
• RQ2 Does maximizing the angular deviation between the defender’s and attacker’s training gradients robustly undermine various model stealing attacks under realistic budgets?
• RQ3 How does MAD compare to existing perturbation-based defenses in terms of non-replicability, utility, and perturbation magnitude?
• RQ4 Is the defense effective across diverse datasets and victim architectures, including stronger DNNs like VGG16 on CIFAR/CUB200?
• RQ5 What countermeasures do attackers plausibly adopt, and how resilient is MAD to such subversions?

Key findings

• MAD consistently reduces attacker performance across all tested datasets and attacks, with defender accuracy remaining close to undefended levels at low perturbation.
• On MNIST, the defense can reduce attacker accuracy by up to about half (e.g., 52% reduction for jbtop3) with less than 1% loss in defender accuracy.
• On datasets like CUB200, the strongest attack’s accuracy drops significantly (e.g., 23% for Knockoff attack) for small defender utility loss (~2%).
• MAD-argmax preserves the top-1 label while still degrading attacker performance, achieving competitive non-replicability with lower perturbation.
• MAD outperforms baseline defenses (reverse-sigmoid, random noise, DP-SGD) in terms of lower perturbation for similar or better non-replicability.
• Ablation studies show using gradient-based G and targeted extreme-point perturbations are crucial; random perturbations perform worse.