[Paper Review] Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

Deep Ganguli, Liane Lovitt+34 more|arXiv (Cornell University)|Aug 23, 2022

Hate Speech and Cyberbullying DetectionComputer Science99 citations

TL;DR

The authors perform manual red teaming across multiple model sizes and safety interventions, release a large red-team attack dataset, and analyze scaling trends and harms to inform safety practices for language models.

ABSTRACT

We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.

Motivation & Objective

• Investigate how red team effectiveness scales across model sizes and safety interventions.
• Quantify the types and frequencies of harmful outputs uncovered by red teaming.
• Provide transparent methodologies, datasets, and guidelines to advance community norms in red teaming.
• Examine limitations and policy implications for safer deployment of language models.

Proposed method

• Red team a general language model to induce harmful outputs via open-ended, topic-focused conversations.
• Compare four model types: Plain LM, HHH-prompted LM, Rejection Sampling (RS), and RLHF-based HH model.
• Collect paired responses (more harmful vs less harmful) to train a harmlessness preference model.
• Measure attack success via red team self-ratings and harmfulness scores from the preference model.
• Aggregate harms using minimum harmlessness score and analyze distributions across model types and sizes.
• Release a dataset of 38,961 red-team attacks and provide accompanying documentation.

Experimental results

Research questions

• RQ1 How does red team difficulty vary with model size and with different safety interventions (Plain LM, HHH prompting, RS, RLHF)?
• RQ2 What are the prevalent types of harms uncovered, and how do they cluster across model types and sizes?
• RQ3 How effective is the Harmlessness Preference Model in guiding safety interventions and reducing harmful outputs?
• RQ4 What are the limitations and ethical considerations in large-scale red teaming with crowdworkers?

Key findings

• RLHF models become increasingly difficult to red team as model size grows.
• Plain LM, HH prompted LM, and RS models show a flat scaling trend with size in red team susceptibility.
• RS models are the most difficult to red team at any scale but tend to be harmless by being evasive.
• RLHF and RS interventions reduce average harmfulness but do not eliminate harmful outputs (visible in distribution tails).
• A dataset of 38,961 red-team attacks is released, enabling broader analysis and tooling development.
• Harm typologies include offensive language, discrimination, harassment, disinformation, and PII-related content; clustering reveals semantically related attack categories.