[Paper Review] Forcing Generative Models to Degenerate Ones: The Power of Data Poisoning Attacks

Shuli Jiang, Swanand Kadhe+3 more|arXiv (Cornell University)|Dec 7, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper investigates data poisoning attacks on large language models (LLMs) fine-tuned for natural language generation (NLG) tasks, demonstrating that as little as 1% of poisoned training data can successfully manipulate model behavior. It introduces novel metrics—Poisoned Target Match and Sneaky Perplexity—to evaluate attack success and stealthiness, showing that trigger length and position significantly impact effectiveness, especially in text completion and summarization tasks using T5-small and GPT-2 models.

ABSTRACT

Growing applications of large language models (LLMs) trained by a third party raise serious concerns on the security vulnerability of LLMs.It has been demonstrated that malicious actors can covertly exploit these vulnerabilities in LLMs through poisoning attacks aimed at generating undesirable outputs. While poisoning attacks have received significant attention in the image domain (e.g., object detection), and classification tasks, their implications for generative models, particularly in the realm of natural language generation (NLG) tasks, remain poorly understood. To bridge this gap, we perform a comprehensive exploration of various poisoning techniques to assess their effectiveness across a range of generative tasks. Furthermore, we introduce a range of metrics designed to quantify the success and stealthiness of poisoning attacks specifically tailored to NLG tasks. Through extensive experiments on multiple NLG tasks, LLMs and datasets, we show that it is possible to successfully poison an LLM during the fine-tuning stage using as little as 1\% of the total tuning data samples. Our paper presents the first systematic approach to comprehend poisoning attacks targeting NLG tasks considering a wide range of triggers and attack settings. We hope our findings will assist the AI security community in devising appropriate defenses against such threats.

Motivation & Objective

• To investigate the feasibility and effectiveness of data poisoning attacks on LLMs fine-tuned for natural language generation (NLG) tasks.
• To address the lack of standardized metrics for evaluating poisoning attack success and stealthiness in NLG, where output space is stochastic and label-flipping is ambiguous.
• To compare the vulnerability of full fine-tuning and parameter-efficient fine-tuning (PEFT), particularly prefix-tuning, to poisoning attacks across different NLG tasks.
• To analyze the impact of trigger characteristics—such as relative length and position—on attack effectiveness in NLG settings.

Proposed method

• Proposes a new evaluation metric, Poisoned Target Match, which measures the proportion of the target output that appears verbatim in the model’s generation, independent of irrelevant context.
• Introduces Sneaky Perplexity as a stealthiness metric, comparing the perplexity of poisoned inputs to clean inputs to assess how closely the model’s behavior mimics the clean model.
• Employs classical data poisoning techniques by injecting inputs with triggers and corresponding attacker-controlled target outputs into the fine-tuning dataset.
• Applies both full fine-tuning and prefix-tuning (a PEFT method) to fine-tune T5-small and GPT-2 models on text summarization and text completion tasks.
• Systematically varies trigger length, position (fixed vs. floating), and insertion method (word-level vs. sentence-level) to study their impact on attack success.
• Uses ROUGE scores and perplexity as baseline metrics, but demonstrates their limitations in capturing true attack success in NLG tasks.

Experimental results

Research questions

• RQ1 Can data poisoning attacks successfully manipulate LLMs fine-tuned for NLG tasks such as text summarization and text completion?
• RQ2 How do trigger length and position affect the success and stealthiness of poisoning attacks in NLG settings?
• RQ3 How do different fine-tuning methods—full fine-tuning versus prefix-tuning—compare in terms of vulnerability to poisoning attacks?
• RQ4 To what extent do standard metrics like ROUGE and perplexity accurately reflect attack success in NLG tasks?
• RQ5 What novel metrics are needed to effectively evaluate the success and stealthiness of poisoning attacks in stochastic, open-ended NLG tasks?

Key findings

• As little as 1% of poisoned training data can successfully manipulate LLMs during fine-tuning, achieving high attack success rates on both text summarization and text completion tasks.
• The Poisoned Target Match metric outperforms ROUGE in capturing true attack success, especially in text completion, where model outputs naturally include context before the target phrase.
• Trigger relative length and position are critical factors: longer triggers and strategically placed triggers (e.g., at sentence boundaries) significantly improve attack success.
• Full fine-tuning is more vulnerable than prefix-tuning in text completion, but less vulnerable in text summarization, indicating task-dependent vulnerability.
• Sneaky Perplexity effectively measures stealthiness, showing that poisoned models maintain perplexity close to clean models, indicating low detectability.
• In the best-case scenario, the attack achieves a Poisoned Target Match of 0.9826 (98.26%) on GPT-2 with 1% poisoned data, while maintaining Sneaky Perplexity within 0.1 of the clean model’s value.