[Paper Review] PoTrojan: powerful neural-level trojan designs in deep learning models

Minhui Zou, Yang Shi+4 more|arXiv (Cornell University)|Feb 8, 2018

Adversarial Robustness in Machine Learning11 references49 citations

TL;DR

This paper introduces PoTrojan, neuron-level trojans that can be inserted into pre-trained neural networks without retraining, to be triggered by rare inputs and cause malfunction or misclassification, demonstrated on AlexNet and VGG16.

ABSTRACT

With the popularity of deep learning (DL), artificial intelligence (AI) has been applied in many areas of human life. Neural network or artificial neural network (NN), the main technique behind DL, has been extensively studied to facilitate computer vision and natural language recognition. However, the more we rely on information technology, the more vulnerable we are. That is, malicious NNs could bring huge threat in the so-called coming AI era. In this paper, for the first time in the literature, we propose a novel approach to design and insert powerful neural-level trojans or PoTrojan in pre-trained NN models. Most of the time, PoTrojans remain inactive, not affecting the normal functions of their host NN models. PoTrojans could only be triggered in very rare conditions. Once activated, however, the PoTrojans could cause the host NN models to malfunction, either falsely predicting or classifying, which is a significant threat to human society of the AI era. We would explain the principles of PoTrojans and the easiness of designing and inserting them in pre-trained deep learning models. PoTrojans doesn't modify the existing architecture or parameters of the pre-trained models, without re-training. Hence, the proposed method is very efficient.

Motivation & Objective

• Introduce the concept of neuron-level trojans (PoTrojans) in pre-trained neural networks.
• Show toy examples and illustrate how PoTrojans can be designed and inserted without retraining.
• Present a general algorithm for crafting triggers and payloads for PoTrojans.
• Validate tacitness and harmfulness of PoTrojans on real-world models (AlexNet and VGG16).

Proposed method

• Define PoTrojan components: trigger synapses and payload synapses.
• Demonstrate single-neuron and multi-neuron PoTrojan designs to create rare triggering conditions.
• Provide trigger design equations to ensure low probability of activation.
• Describe payload design to steer outputs after activation, with and without access to training instances.
• Outline an algorithm (including gradient-based steps) to compute triggering and payload parameters.
• Experimentally validate on AlexNet and VGG16 trained on ImageNet to assess triggering and impact.

Experimental results

Research questions

• RQ1 Can neuron-level trojans be inserted into pre-trained models without modifying architecture or retraining?
• RQ2 How can triggers be designed to activate PoTrojans under extremely rare conditions?
• RQ3 How should the payloads be crafted to meaningfully alter model outputs when triggered?
• RQ4 What is the tacitness (accidental triggering rate) and harmful impact of PoTrojans in real networks like AlexNet and VGG16?

Key findings

• PoTrojans can be inserted in pre-trained models without changing existing parameters or architecture, enabling efficient exploitation.
• Trigger inputs can activate PoTrojans with high reliability when present (examples show 100% triggering in tested insertions).
• Accidental triggering on non-trigger inputs is negligible in experiments (accident triggering rate reported as 0 for tested PoTrojans).
• Triggered PoTrojans can cause significant misclassification or targeted outputs, including forcing specific labels with high confidence in tested cases.
• Payload design allows steering outputs toward a chosen target (e.g., mislabeling input {1,1,1,1} as a specific label with high probability).
• Experiments on AlexNet and VGG16 demonstrate both triggering behavior and impactful outputs under trigger conditions.