[Paper Review] Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs

Yuxia Wang, Haonan Li+3 more|arXiv (Cornell University)|Aug 25, 2023

Software Engineering Research4 citations

TL;DR

This paper introduces Do-Not-Answer, the first open-source dataset for evaluating safeguards in large language models (LLMs), consisting of 939 risky instructions that responsible models should not answer. It proposes automatic safety evaluation using small BERT-like classifiers trained on human-annotated responses, achieving performance comparable to GPT-4 at low cost.

ABSTRACT

With the rapid evolution of large language models (LLMs), new and hard-to-predict harmful capabilities are emerging. This requires developers to be able to identify risks through the evaluation of "dangerous capabilities" in order to responsibly deploy LLMs. In this work, we collect the first open-source dataset to evaluate safeguards in LLMs, and deploy safer open-source LLMs at a low cost. Our dataset is curated and filtered to consist only of instructions that responsible language models should not follow. We annotate and assess the responses of six popular LLMs to these instructions. Based on our annotation, we proceed to train several BERT-like classifiers, and find that these small classifiers can achieve results that are comparable with GPT-4 on automatic safety evaluation. Warning: this paper contains example data that may be offensive, harmful, or biased.

Motivation & Objective

• To address the growing risk of emergent harmful capabilities in LLMs, especially in open-source models lacking robust safety mechanisms.
• To create a standardized, open dataset of risky instructions that responsible LLMs should refuse to answer.
• To develop low-cost, automated evaluation methods for assessing LLM safety using small, fine-tuned classifiers.
• To benchmark the safety performance of both commercial and open-source LLMs across a fine-grained risk taxonomy.
• To enable scalable, reliable safety evaluation without relying on expensive models like GPT-4.

Proposed method

• Proposed a three-level hierarchical risk taxonomy covering five distinct harm types, with at least ten prompts per category, resulting in 939 total risky instructions.
• Collected and manually annotated over 5,000 model responses from six LLMs (including GPT-4, LLaMA-2, Vicuna, and ChatGLM2) to these instructions.
• Trained BERT-like classifiers on the annotated responses to perform automatic action classification and harmful response detection.
• Used both instruction-response pairs and response-only inputs to evaluate the impact of context on classification performance.
• Evaluated the effectiveness of Longformer (2048-token context) versus BERT (512-token context) to assess the role of context length in detecting complex harmful behaviors.
• Benchmarked the performance of the small classifiers against GPT-4 as a human-like evaluator, using accuracy and macro-F1 as metrics.

Figure 1: A comprehensive evaluation of LLM safeguards.

Experimental results

Research questions

• RQ1 Can a small, fine-tuned BERT-like model achieve safety evaluation performance comparable to GPT-4?
• RQ2 How effective are small models in detecting harmful responses when given both instructions and responses as input?
• RQ3 Does longer context length (e.g., Longformer) improve detection of complex harmful behaviors compared to shorter-context models?
• RQ4 How do different LLMs (commercial and open-source) vary in their refusal rates and response patterns to dangerous instructions?
• RQ5 To what extent do multi-label response annotations improve the accuracy of safety evaluation compared to single-label classification?

Key findings

• Fine-tuned BERT-like models with under 600 million parameters achieved over 98% accuracy and 80% macro-F1 in harmful response detection, matching GPT-4's performance.
• LLaMA-2 demonstrated the highest refusal rate (99.7%) among the evaluated models, while ChatGLM2 had the lowest (90.9%).
• The inclusion of instructions as input improved classification performance, especially for action classification, compared to using responses alone.
• Longformer outperformed BERT in detecting complex harmful behaviors, particularly in categories requiring full response context (e.g., category 2 and 5).
• The ranking of models by safety performance was nearly identical between GPT-4 and the Longformer-based evaluator, confirming the validity of the automatic evaluation method.
• The study revealed distinct risk-type-specific response patterns, indicating that models exhibit predictable failure modes under specific types of harmful prompts.

Figure 2: Three-level taxonomy of LLM risks.