[Paper Review] PerD: Perturbation Sensitivity-based Neural Trojan Detection Framework on NLP Applications

Diego Garcia-soto, Huili Chen+1 more|arXiv (Cornell University)|Aug 8, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes PerD, a model-level neural Trojan detection framework for NLP applications that analyzes perturbation sensitivity by injecting Robustness-Aware Perturbations (RAP) into inputs to extract model response signatures. It uses a meta-classifier trained on these signatures to detect Trojans with high accuracy across sentiment classification, NER, and QA tasks, achieving up to 100% AUC on TrojAI datasets and enabling a lightweight black-box variant with relaxed RAP triggers that reduces detection time while preserving performance.

ABSTRACT

Deep Neural Networks (DNNs) have been shown to be susceptible to Trojan attacks. Neural Trojan is a type of targeted poisoning attack that embeds the backdoor into the victim and is activated by the trigger in the input space. The increasing deployment of DNNs in critical systems and the surge of outsourcing DNN training (which makes Trojan attack easier) makes the detection of Trojan attacks necessary. While Neural Trojan detection has been studied in the image domain, there is a lack of solutions in the NLP domain. In this paper, we propose a model-level Trojan detection framework by analyzing the deviation of the model output when we introduce a specially crafted perturbation to the input. Particularly, we extract the model's responses to perturbed inputs as the `signature' of the model and train a meta-classifier to determine if a model is Trojaned based on its signature. We demonstrate the effectiveness of our proposed method on both a dataset of NLP models we create and a public dataset of Trojaned NLP models from TrojAI. Furthermore, we propose a lightweight variant of our detection method that reduces the detection time while preserving the detection rates.

Motivation & Objective

• To address the growing threat of backdoor attacks in NLP models, especially with increasing outsourcing of model training.
• To develop a model-level detection method that identifies Trojans without requiring access to model parameters or input triggers.
• To enable effective detection across diverse NLP tasks such as sentiment classification, NER, and question answering.
• To reduce detection time while maintaining high detection rates through a lightweight variant using relaxed RAP triggers.
• To guide defenders in selecting between architecture-agnostic and architecture-specific meta-classifiers based on histogram distribution similarity across model architectures.

Proposed method

• The framework injects Robustness-Aware Perturbations (RAP) into input samples to elicit model response deviations, forming a signature of the model’s sensitivity to perturbations.
• It constructs a histogram of confidence score deviations across multiple perturbed inputs to represent the model’s response signature.
• A meta-classifier is trained to distinguish between benign and Trojaned models based on their response signatures.
• Data augmentation is applied by incorporating inputs from public datasets (e.g., IMDB, Squad_v2) to improve signature quality and detection precision.
• A lightweight variant replaces optimized RAP triggers with random noise patterns, enabling black-box operation without model parameter access.
• The method supports both architecture-agnostic and architecture-specific meta-classifiers, with selection guided by histogram similarity across model architectures.

Experimental results

Research questions

• RQ1 Can perturbation sensitivity be effectively leveraged to detect neural Trojans in NLP models without requiring access to model parameters or input triggers?
• RQ2 How does data augmentation using public datasets improve the precision of model response signatures for Trojan detection?
• RQ3 What is the performance trade-off between detection accuracy and efficiency when using relaxed RAP triggers instead of optimized ones?
• RQ4 Under what conditions is an architecture-agnostic meta-classifier superior to architecture-specific ones in detecting Trojans?
• RQ5 How does the distribution of response deviation histograms vary across different NLP model architectures, and how does this affect meta-classifier design?

Key findings

• On the TrojAI sentiment classification dataset, PerD achieved 95.8% accuracy and 100% AUC using architecture-specific meta-classifiers with RAP triggers.
• The lightweight variant with relaxed RAP triggers reduced detection time significantly while maintaining high performance, achieving 94.9% accuracy and 100% AUC on the same task.
• Data augmentation using public datasets like IMDB improved detection performance, especially when combined with RAP-based signatures.
• For question-answering models, the architecture-agnostic meta-classifier outperformed architecture-specific ones, as their response histograms were highly similar across architectures.
• On named entity recognition, architecture-specific meta-classifiers achieved better results than the architecture-agnostic version, indicating that histogram distributions varied significantly across model types.
• The study provides empirical guidance: when histogram distributions differ across architectures, use architecture-specific meta-classifiers; otherwise, a unified meta-classifier performs better.