[Paper Review] Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts

Mikayel Samvelyan, Sharath Chandra Raparthy+10 more|arXiv (Cornell University)|Feb 26, 2024

Advanced Malware Detection Techniques7 citations

TL;DR

Rainbow Teaming uses quality-diversity search (MAP-Elites) to automatically generate a diverse archive of adversarial prompts for LLMs, improving safety and enabling synthetic data for robustness without sacrificing general capabilities.

ABSTRACT

As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to adversarial attacks is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming, a novel black-box approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem and uses open-ended search to generate prompts that are both effective and diverse. Focusing on the safety domain, we use Rainbow Teaming to target various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90% across all tested models. Furthermore, we demonstrate that prompts generated by Rainbow Teaming are highly transferable and that fine-tuning models with synthetic data generated by our method significantly enhances their safety without sacrificing general performance or helpfulness. We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity, showcasing its potential to drive robust open-ended self-improvement in a wide range of applications.

Motivation & Objective

• Motivate robust evaluation of LLM safety across diverse attack vectors and domains.
• Develop a general, open-ended method to generate diverse adversarial prompts without heavy human input.
• Show that diversity improves diagnostic coverage and enables effective synthetic data for safety Fine-Tuning (SFT).
• Demonstrate cross-domain applicability (safety, QA, cybersecurity) and transferability across model sizes.

Proposed method

• Cast adversarial prompt generation as a quality-diversity (QD) problem using MAP-Elites.
• Construct a K-dimensional feature archive that encodes diversity (e.g., Risk Category, Attack Style).
• Use a Mutator LLM to generate candidate prompts conditioned on a prescribed feature descriptor.
• Query a Target LLM with candidate prompts and a Judge LLM to compare safety/unsafe responses and update the archive.
• Employ a Judge-based preference model to avoid reward hacking and promote open-ended improvement.
• Optionally apply domain-specific mutations and evaluate with domain-relevant evaluators (GPT-4, Llama Guard).

Experimental results

Research questions

• RQ1 Can open-ended QD search generate a broad, high-quality archive of adversarial prompts across safety, QA, and cybersecurity domains?
• RQ2 Do adversarial prompts discovered for one model or domain transfer to other models or domains?
• RQ3 Does incorporating a similarity filter preserve prompt diversity without sacrificing effectiveness?
• RQ4 Do system prompts and preference models significantly influence robustness outcomes and evaluation bias?
• RQ5 Can synthetic data produced by Rainbow Teaming meaningfully improve safety and robustness when used for fine-tuning?

Key findings

• The method discovers hundreds of adversarial prompts per domain/model in 2000 iterations, enabling diverse vulnerability diagnostics.
• In safety experiments on Llama 2-chat variants, 7B reached ~92% ASR (GPT-4) and 84% (13B) depending on model, with 70B around 87% (GPT-4).
• Transfer across model sizes is substantial, e.g., prompts generated for 7B transfer to 13B and 70B at notable rates (46% and 53% for respective targets).
• A similarity filter at mutation stage maintains diversity and reduces self-BLEU from 0.90 to 0.39, while preserving high ASR (GPT-4 0.92, Llama Guard 0.89).
• Comparative Judge-based preference avoids reward hacking and yields higher GPT-4 ASR alignment than score-based approaches.
• Fine-tuning on synthetic data produced by Rainbow Teaming dramatically reduces ASR (7B: GPT-4 from 0.92 to 0.026; Llama Guard from 0.82 to 0.013) without harming GSM8K or MMLU; further rounds of adversarial training improve robustness.