[Paper Review] Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples

Minhao Cheng, Jinfeng Yi+3 more|arXiv (Cornell University)|Mar 3, 2018

Adversarial Robustness in Machine LearningComputer Science23 references69 citations

TL;DR

Seq2Sick presents an optimization-based attack on seq2seq models to craft adversarial inputs that cause targeted or non-overlapping output changes, using projected gradient methods, group lasso, and gradient regularization.

ABSTRACT

Crafting adversarial examples has become an important technique to evaluate the robustness of deep neural networks (DNNs). However, most existing works focus on attacking the image classification problem since its input space is continuous and output space is finite. In this paper, we study the much more challenging problem of crafting adversarial examples for sequence-to-sequence (seq2seq) models, whose inputs are discrete text strings and outputs have an almost infinite number of possibilities. To address the challenges caused by the discrete input space, we propose a projected gradient method combined with group lasso and gradient regularization. To handle the almost infinite output space, we design some novel loss functions to conduct non-overlapping attack and targeted keyword attack. We apply our algorithm to machine translation and text summarization tasks, and verify the effectiveness of the proposed algorithm: by changing less than 3 words, we can make seq2seq model to produce desired outputs with high success rates. On the other hand, we recognize that, compared with the well-evaluated CNN-based classifiers, seq2seq models are intrinsically more robust to adversarial attacks.

Motivation & Objective

• Motivate robustness evaluation for seq2seq models in safety-critical NLP tasks.
• Develop an optimization framework to generate adversarial inputs under discrete input constraints.
• Address the large, almost infinite output space with targeted and non-overlapping output attacks.
• Propose techniques to handle discrete inputs and promote sparse, meaningful perturbations.
• Assess how seq2seq robustness compares to CNN-based image classifiers.

Proposed method

• Formulate adversarial attack as min_delta { L(X+delta) + lambda R(delta) } with R being a group lasso penalty.
• Use projected gradient descent with gradient regularization to keep perturbations within the input vocabulary space.
• Design non-overlapping attack loss L_non-overlapping to force output words to differ from the original at every position.
• Design targeted keywords attack loss L_keywords to ensure targeted keywords appear in the output, with a mask to avoid keyword collisions.
• Enforce X+delta ∈ W (input vocabulary) via projection; apply grouped sparsity to perturb only a subset of input words.
• Incorporate gradient regularization term to encourage proximity to the embedding space and enable feasible word mappings.

Experimental results

Research questions

• RQ1 Can seq2seq models be meaningfully attacked with small, sparse input changes to induce large output changes?
• RQ2 Are seq2seq models more robust to adversarial manipulation than CNN-based image classifiers?
• RQ3 How can discrete input constraints and almost infinite output spaces be effectively handled in adversarial attacks on seq2seq models?
• RQ4 What is the impact of targeted keyword attacks on translation and summarization outputs?

Key findings

• Seq2Sick achieves high success rates for non-overlapping attacks and targeted keyword attacks with only 1–3 word changes.
• Non-overlapping attack success rates: text summarization Gigaword 86.0%, DUC2003 85.2%, DUC2004 84.2% with BLEU scores around 0.77–0.83.
• Targeted keyword attack shows high success with 1 keyword and lower success as keywords increase (e.g., Gigaword 1-keyword 99.8% success, BLEU 0.801; 3-keyword 43.0%).
• Machine translation non-overlap success 89.4%; 1-keyword 100.0%; 2-keyword 91.0%; 3-keyword 69.6% with BLEU dropping as keywords increase.
• Adversarial examples preserve semantic meaning for most cases (sentiment test: 2.2% semantic change).
• Seq2seq models exhibit intrinsic robustness relative to CNN classifiers due to discrete input and exponentially large output space.