[Paper Review] Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers

Ji Gao, Jack Lanchantin+2 more|arXiv (Cornell University)|Jan 13, 2018

Adversarial Robustness in Machine LearningComputer Science23 references57 citations

TL;DR

The paper presents DeepWordBug, a black-box method to generate small, imperceptible text perturbations that cause deep learning classifiers to misclassify, by scoring token importance and applying character-level transformations.

ABSTRACT

Although various techniques have been proposed to generate adversarial samples for white-box attacks on text, little attention has been paid to black-box attacks, which are more realistic scenarios. In this paper, we present a novel algorithm, DeepWordBug, to effectively generate small text perturbations in a black-box setting that forces a deep-learning classifier to misclassify a text input. We employ novel scoring strategies to identify the critical tokens that, if modified, cause the classifier to make an incorrect prediction. Simple character-level transformations are applied to the highest-ranked tokens in order to minimize the edit distance of the perturbation, yet change the original classification. We evaluated DeepWordBug on eight real-world text datasets, including text classification, sentiment analysis, and spam detection. We compare the result of DeepWordBug with two baselines: Random (Black-box) and Gradient (White-box). Our experimental results indicate that DeepWordBug reduces the prediction accuracy of current state-of-the-art deep-learning models, including a decrease of 68\% on average for a Word-LSTM model and 48\% on average for a Char-CNN model.

Motivation & Objective

• Demonstrate a black-box adversarial attack on NLP classifiers that does not require model parameters or gradients.
• Identify important tokens that drive classifier decisions using model-agnostic scoring functions.
• Apply small, imperceptible character-level perturbations to those tokens to induce misclassification.
• Evaluate attack effectiveness across multiple datasets and model architectures (Word-LSTM and Char-CNN).
• Assess transferability of adversarial samples between models and robustness to parameter choices.

Proposed method

• Define four token-scoring functions (Replace-1, Temporal Head, Temporal Tail, Combined) to rank token importance without model parameters.
• Rank tokens by importance using model outputs when replacing a token with an out-of-vocabulary token or by sequential prediction differences.
• Select the top m tokens and perturb them with simple, small character-level transforms (swap, substitution, deletion, insertion) to create adversarial samples.
• Use Levenshtein edit distance as the perturbation constraint to keep changes perceptually small.
• Attack algorithm DeepWordBug operates in O(n) queries relative to input length.

Experimental results

Research questions

• RQ1 Can a black-box adversarial attack significantly reduce NLP model accuracy without gradients or model internals?
• RQ2 Do adversarial samples generated for one model transfer effectively to other models on the same task?
• RQ3 How robust are DeepWordBug’s scoring and transformation strategies to different dictionary sizes and transform choices?
• RQ4 What is the effectiveness of DeepWordBug across diverse text datasets and architectures (word-level LSTM and character-level CNN)?

Key findings

• DeepWordBug reduces accuracy by large margins across eight datasets, achieving a 68% average decrease for Word-LSTM and 48% average decrease for Char-CNN under perturbation budget ϵ = 30.
• Combined scoring with Substitution transformer dramatically lowers AG’s News and Amazon Review Polarity accuracies for Word-LSTM (e.g., from ~90% to ~25% on AG’s News).
• Replace-1 scoring with Substitution transformer is highly effective for Char-CNN (e.g., ~90% to ~46% on Amazon Review Polarity).
• Attack samples transfer well between models, lowering accuracy from ~90% to 20–50% on alternative architectures and embeddings.
• Random scoring performs poorly; Gradient (white-box) baselines are outperformed by the proposed scoring functions, highlighting the value of model-agnostic token importance.