[Paper Review] MART: Improving LLM Safety with Multi-round Automatic Red-Teaming

Suyu Ge, Chunting Zhou+6 more|arXiv (Cornell University)|Nov 13, 2023

Topic Modeling4 citations

TL;DR

MART proposes a multi-round automatic red-teaming framework that iteratively improves LLM safety by pitting an adversarial LLM against a target LLM, where both models evolve through adversarial competition: the adversarial LLM generates increasingly effective attacks, while the target LLM is fine-tuned on safe responses to these attacks. After four rounds, MART reduces violation rates by up to 84.7% on adversarial benchmarks, matching the safety of models with extensive manual red-teaming, while preserving model helpfulness.

ABSTRACT

Red-teaming is a common practice for mitigating unsafe behaviors in Large Language Models (LLMs), which involves thoroughly assessing LLMs to identify potential flaws and addressing them with responsible and accurate responses. While effective, manual red-teaming is costly, and existing automatic red-teaming typically discovers safety risks without addressing them. In this paper, we propose a Multi-round Automatic Red-Teaming (MART) method, which incorporates both automatic adversarial prompt writing and safe response generation, significantly increasing red-teaming scalability and the safety of the target LLM. Specifically, an adversarial LLM and a target LLM interplay with each other in an iterative manner, where the adversarial LLM aims to generate challenging prompts that elicit unsafe responses from the target LLM, while the target LLM is fine-tuned with safety aligned data on these adversarial prompts. In each round, the adversarial LLM crafts better attacks on the updated target LLM, while the target LLM also improves itself through safety fine-tuning. On adversarial prompt benchmarks, the violation rate of an LLM with limited safety alignment reduces up to 84.7% after 4 rounds of MART, achieving comparable performance to LLMs with extensive adversarial prompt writing. Notably, model helpfulness on non-adversarial prompts remains stable throughout iterations, indicating the target LLM maintains strong performance on instruction following.

Motivation & Objective

• To address the high cost and limited scalability of manual red-teaming in LLM safety evaluation.
• To overcome the static nature of existing automatic red-teaming methods that do not adapt to evolving model vulnerabilities.
• To develop a scalable, automated framework that simultaneously improves both adversarial prompt generation and target model safety through iterative interaction.
• To maintain model helpfulness on non-adversarial tasks during safety alignment.
• To reduce reliance on human annotators in red-teaming by enabling model-based, self-improving safety evaluation.

Proposed method

• An adversarial LLM is trained to generate increasingly effective malicious prompts by leveraging feedback from previous successful attacks.
• The target LLM generates responses to these adversarial prompts, which are then evaluated using a safety reward model to identify unsafe outputs.
• Prompts that successfully trigger unsafe responses are used to fine-tune the adversarial LLM in subsequent rounds, improving its attack capability.
• Safe and high-quality responses from the target LLM are collected and used to fine-tune the target model itself, enhancing its safety alignment.
• The process iterates over multiple rounds, with both models improving through adversarial competition.
• Additional techniques like context distillation and rejection sampling are applied to enrich training data diversity and quality at specific rounds.

Figure 1: Illustration of Mart . On the left figure, according to the feedback from the evaluator, Mart first identifies successful attacks from generated prompts, and then leverages them to train the adversarial LLM $\mathcal{M}_{adv}$ . On the contrary, the right figure illustrates a successful de

Experimental results

Research questions

• RQ1 Can an automated, iterative red-teaming framework effectively improve LLM safety without relying on human red-teamers?
• RQ2 Does continuous adversarial prompting adapt to evolving model vulnerabilities, enabling sustained safety improvement?
• RQ3 Can safety alignment be achieved without degrading model performance on instruction-following and helpfulness tasks?
• RQ4 To what extent can automatic red-teaming reduce violation rates compared to manual or baseline automatic methods?
• RQ5 How does the integration of adversarial prompt generation and safe response generation jointly enhance model robustness?

Key findings

• After four rounds of MART, the violation rate of a minimally safety-aligned LLM decreased by up to 84.7% on adversarial prompt benchmarks.
• The safety improvement achieved by MART is comparable to that of models fine-tuned with extensive manual red-teaming, such as ChatGPT.
• Model helpfulness on non-adversarial instruction-following benchmarks remained stable throughout all rounds, indicating no degradation in core capabilities.
• The method effectively adapts to evolving model vulnerabilities by generating increasingly sophisticated adversarial prompts across rounds.
• The integration of rejection sampling and context distillation further improved the diversity and quality of adversarial training data.
• Human evaluation showed a 53.7% reduction in unsafe generations, confirming the real-world impact of the safety alignment.

(a) Safety trend on our SafeEval