[Paper Review] Adversarial Attack and Defense of Structured Prediction Models

Wenjuan Han, Liwen Zhang+2 more|arXiv (Cornell University)|Oct 4, 2020

Adversarial Robustness in Machine Learning34 references4 citations

TL;DR

This paper proposes a novel, black-box, online adversarial attack framework for structured prediction models in NLP using a sequence-to-sequence generator trained via reinforcement learning with feedback from multiple reference models. The method generates fluent, highly adversarial sentences that effectively fool state-of-the-art dependency parsers and POS taggers, and it further improves model robustness through adversarial training, achieving up to 80.1% sentence-level attack success rate and significant fluency gains.

ABSTRACT

Building an effective adversarial attacker and elaborating on countermeasures for adversarial attacks for natural language processing (NLP) have attracted a lot of research in recent years. However, most of the existing approaches focus on classification problems. In this paper, we investigate attacks and defenses for structured prediction tasks in NLP. Besides the difficulty of perturbing discrete words and the sentence fluency problem faced by attackers in any NLP tasks, there is a specific challenge to attackers of structured prediction models: the structured output of structured prediction models is sensitive to small perturbations in the input. To address these problems, we propose a novel and unified framework that learns to attack a structured prediction model using a sequence-to-sequence model with feedbacks from multiple reference models of the same structured prediction task. Based on the proposed attack, we further reinforce the victim model with adversarial training, making its prediction more robust and accurate. We evaluate the proposed framework in dependency parsing and part-of-speech tagging. Automatic and human evaluations show that our proposed framework succeeds in both attacking state-of-the-art structured prediction models and boosting them with adversarial training.

Motivation & Objective

• To address the challenge of attacking structured prediction models in NLP, which are highly sensitive to input perturbations and differ fundamentally from classification tasks.
• To overcome limitations of gradient-based and word-level attack methods that fail due to discrete token constraints and sentence fluency issues.
• To design a black-box, online attacker that does not require model gradients or iterative optimization, enabling efficient and generalizable attacks.
• To develop a defense mechanism via adversarial training using the generated adversarial examples to improve model robustness.

Proposed method

• A sequence-to-sequence (seq2seq) sentence generator is trained using reinforcement learning with a custom reward function that evaluates the victim model’s output against an ensemble of multiple reference models for the same structured prediction task.
• The reward function combines three components: the likelihood of the victim model’s output, the similarity between the victim and reference models’ outputs, and sentence fluency scores from pre-trained language models.
• The attacker operates in a black-box setting, requiring no access to the victim model’s architecture, parameters, or gradients, enabling broad applicability.
• The attack is online: once trained, the generator produces adversarial examples directly from input sentences without further optimization or model access.
• Adversarial training is applied by fine-tuning the victim model on the generated adversarial examples, improving its robustness and accuracy.
• The framework is evaluated on dependency parsing and POS tagging using both automatic and human evaluations.

Experimental results

Research questions

• RQ1 Can a seq2seq-based generator with multi-reference feedback produce adversarial examples that effectively attack structured prediction models in NLP?
• RQ2 How does the choice of reference models (identical vs. different) affect the fluency and adversarial effectiveness of generated examples?
• RQ3 Can the proposed attack framework be applied effectively across different structured prediction models and architectures?
• RQ4 To what extent does adversarial training using the generated examples improve the robustness and accuracy of the victim models?
• RQ5 How does the proposed method compare to existing gradient-based or word-level attack methods in terms of attack success and sentence fluency?

Key findings

• The proposed attack framework achieved a sentence-level attacking success rate of 80.1% on dependency parsing using the Deep Biaffine and BiST parsers as references, significantly outperforming baseline methods.
• Human evaluation showed that the generated adversarial sentences had a fluency score of 3.84 (on a 5-point scale), outperforming both AllSame (4.19) and EvalSame (3.54) settings, indicating better balance between fluency and attack effectiveness.
• The attack framework successfully generated adversarial examples that caused state-of-the-art structured prediction models to produce incorrect parse trees or POS tags, even when the perturbations were minimal and fluent.
• Adversarial training using the generated examples improved the robustness and accuracy of the victim models, demonstrating the effectiveness of the defense mechanism.
• The framework is generalizable: it achieved consistent performance across different victim models (e.g., StackPtr) and reference parser combinations, confirming its broad applicability.