[Paper Review] The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models

Kiran Karra, Chace Ashcraft+1 more|arXiv (Cornell University)|Mar 13, 2020

Adversarial Robustness in Machine Learning17 references20 citations

TL;DR

The TrojAI software framework is an open-source Python toolkit that enables scalable, reproducible generation of trojaned deep learning models by poisoning datasets with customizable triggers. It demonstrates effective trojan embedding in MNIST classifiers and a reinforcement learning agent, showing that trigger type, batch size, and poisoning rate significantly affect success, with Neural Cleanse detecting anomalies in only 18% of cases.

ABSTRACT

In this paper, we introduce the TrojAI software framework, an open source set of Python tools capable of generating triggered (poisoned) datasets and associated deep learning (DL) models with trojans at scale. We utilize the developed framework to generate a large set of trojaned MNIST classifiers, as well as demonstrate the capability to produce a trojaned reinforcement-learning model using vector observations. Results on MNIST show that the nature of the trigger, training batch size, and dataset poisoning percentage all affect successful embedding of trojans. We test Neural Cleanse against the trojaned MNIST models and successfully detect anomalies in the trained models approximately $18\%$ of the time. Our experiments and workflow indicate that the TrojAI software framework will enable researchers to easily understand the effects of various configurations of the dataset and training hyperparameters on the generated trojaned deep learning model, and can be used to rapidly and comprehensively test new trojan detection methods.

Motivation & Objective

• To address the lack of standardization and reproducibility in trojan attack research by providing a unified, configurable framework for generating poisoned datasets and trojaned models.
• To enable systematic study of how trigger design, poisoning rate, and training hyperparameters affect trojan embedding success and detectability.
• To support rapid evaluation and benchmarking of new trojan detection and mitigation techniques across diverse data modalities and model architectures.
• To extend reproducible research in trojan attacks beyond image classification to include reinforcement learning with vector-observation environments.
• To facilitate future research on stealthier trojan embedding methods and broader data modality support (e.g., audio, object detection).

Proposed method

• The framework consists of two core submodules: datagen for generating poisoned datasets with user-defined triggers, and modelgen for training deep learning models on these datasets.
• It supports configurable trigger types (e.g., static, dynamic, rotated), placement strategies, and poisoning percentages across input data.
• For reinforcement learning, the framework integrates with OpenAI Gym environments, allowing the injection of triggers into vector-based observations during training.
• The framework enables batch training of multiple models with varying hyperparameters (e.g., batch size, epochs) to assess robustness and yield.
• It supports evaluation via standard metrics such as accuracy on clean and triggered data, and cumulative reward in RL tasks.
• Neural Cleanse is used as a detection baseline to evaluate the detectability of trojaned models across different configurations.

Experimental results

Research questions

• RQ1 How do variations in trigger type (static vs. dynamic) and placement affect the success rate of trojan embedding in MNIST classifiers?
• RQ2 What is the impact of dataset poisoning percentage and batch size on the model yield and robustness of trojaned models?
• RQ3 Can the TrojAI framework effectively generate functional trojaned reinforcement learning agents using vector observations?
• RQ4 How well does the Neural Cleanse detection method perform across diverse trojan configurations in terms of anomaly detection rate?
• RQ5 To what extent do results from one trigger or configuration generalize to others in terms of detectability and model performance?

Key findings

• The TrojAI framework successfully generated trojaned MNIST classifiers with high clean accuracy (>95%) and targeted misclassification on triggered inputs, demonstrating effective and scalable trojan injection.
• Model yield and detection performance varied significantly with trigger type and configuration, with dynamic triggers showing lower detection rates by Neural Cleanse compared to static ones.
• Batch size had a measurable impact on training convergence, with larger batches (64) requiring fewer epochs on average (17.9±2.9) than smaller ones (16: 14.1±2.4).
• Neural Cleanse detected trojaned models with approximately 18% success rate across all configurations, indicating limited generalization of detection performance.
• In reinforcement learning, the Boxing agent maintained high performance (52.22 avg. reward) on clean environments and low performance (-62.69) on poisoned ones, confirming successful trojan injection.
• The framework’s modular design enables consistent, reproducible experiments across diverse data modalities, hyperparameters, and model architectures, supporting large-scale benchmarking.