[Paper Review] Is poisoning a real threat to LLM alignment? Maybe more so than you think

Pankayaraj Pathmanathan, Souradip Chakraborty+3 more|arXiv (Cornell University)|Jun 17, 2024

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper investigates the vulnerability of Direct Preference Optimization (DPO) in Large Language Model (LLM) alignment to data poisoning attacks, demonstrating that backdoor attacks can be successfully executed with as little as 0.5% of training data poisoned—significantly lower than the 3–4% required for PPO-based methods. The attack leverages DPO scores and influence analysis to identify high-impact training examples, revealing a critical security gap in DPO’s supervised learning formulation.

ABSTRACT

Recent advancements in Reinforcement Learning with Human Feedback (RLHF) have significantly impacted the alignment of Large Language Models (LLMs). The sensitivity of reinforcement learning algorithms such as Proximal Policy Optimization (PPO) has led to new line work on Direct Policy Optimization (DPO), which treats RLHF in a supervised learning framework. The increased practical use of these RLHF methods warrants an analysis of their vulnerabilities. In this work, we investigate the vulnerabilities of DPO to poisoning attacks under different scenarios and compare the effectiveness of preference poisoning, a first of its kind. We comprehensively analyze DPO's vulnerabilities under different types of attacks, i.e., backdoor and non-backdoor attacks, and different poisoning methods across a wide array of language models, i.e., LLama 7B, Mistral 7B, and Gemma 7B. We find that unlike PPO-based methods, which, when it comes to backdoor attacks, require at least 4\% of the data to be poisoned to elicit harmful behavior, we exploit the true vulnerabilities of DPO more simply so we can poison the model with only as much as 0.5\% of the data. We further investigate the potential reasons behind the vulnerability and how well this vulnerability translates into backdoor vs non-backdoor attacks.

Motivation & Objective

• To investigate the susceptibility of DPO-based LLM alignment to training-time poisoning attacks, particularly in comparison to PPO-based methods.
• To analyze the effectiveness of different poisoning strategies—backdoor and non-backdoor attacks—on DPO under various data selection criteria.
• To evaluate the transferability of influential training points across different LLM architectures (Llama 7B, Mistral 7B, Gemma 7B) for potential black-box attacks.
• To understand why DPO is more vulnerable than PPO to selective poisoning despite its simpler, supervised learning formulation.
• To explore the feasibility of using influence analysis and DPO scores to construct highly effective, low-data poisoning attacks.

Proposed method

• Proposes a white-box poisoning framework that selects training examples based on their DPO score, enabling targeted manipulation of the alignment objective.
• Introduces three data selection strategies: random, DPO-score-based, and gradient-projection-based, to evaluate poisoning efficacy across models.
• Employs influence function analysis to identify training points with maximal impact on model behavior, enabling efficient attack construction.
• Uses a DPO score-based method that maximizes the difference in preference scores between chosen and rejected responses to amplify attack effects.
• Evaluates attacks on three LLMs (Llama 2 7B, Mistral 7B, Gemma 7B) using both backdoor triggers and non-backdoor instruction manipulation.
• Measures attack success via targeted harmful response generation and evaluates transferability of influential points across models.

Figure 1: Overview of the analysis on DPO’s vulnerabilities. We consider two types of attacks (Backdoor, Non Backdoor). When performing these attacks we poison the model using 4 different poisoning methods namely random, DPO score based (Section. 3.2 ), DPO score + semantic diversity based (Section

Experimental results

Research questions

• RQ1 Can DPO-based alignment be effectively poisoned with a minimal fraction of training data, and how does this compare to PPO-based methods?
• RQ2 How does the DPO score-based data selection strategy compare to random or gradient-based methods in terms of poisoning efficiency?
• RQ3 What is the transferability of influential training points across different LLM architectures for potential black-box attacks?
• RQ4 Why is DPO more vulnerable to selective poisoning than PPO, despite PPO’s more complex two-stage training process?
• RQ5 To what extent can non-backdoor attacks be executed with selective poisoning, and what data proportions are required for success?

Key findings

• Backdoor attacks on DPO can be successfully executed with only 0.5% of the training data poisoned, significantly lower than the 3–4% required for PPO-based methods.
• The DPO score-based poisoning method outperforms random and gradient-projection-based methods, achieving high attack success with minimal data injection.
• Influence points identified via DPO scores show measurable overlap across models—especially between Mistral 7B and Gemma 7B—enabling potential transferable attacks.
• Non-backdoor attacks require up to 25% of the data to be poisoned even with selective influence-based sampling, indicating their impracticality in real-world settings.
• The DPO score-based method achieves performance comparable to full gradient-based attacks, suggesting it exploits inherent vulnerabilities in the DPO objective function.
• Despite its simplicity, DPO’s supervised learning formulation introduces a critical security weakness not present in PPO, making it more susceptible to targeted, low-data poisoning.

Figure 2: Four types of poisoning are covered in this work. All of the methods except for random poisoning get a white box feedback from the LLM trained on the non-poisoned, clean data and use the information from those fine-tuned models (DPO score, DPO gradient) to choose points in a selective mann