[Paper Review] Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity

Terry Yue Zhuo, Yujin Huang+2 more|arXiv (Cornell University)|Jan 30, 2023

Adversarial Robustness in Machine Learning170 citations

TL;DR

The paper conducts a qualitative red-teaming of ChatGPT across bias, reliability, robustness, and toxicity, using benchmarks and case studies, and identifies ethical risks and jailbreak vulnerabilities.

ABSTRACT

Recent breakthroughs in natural language processing (NLP) have permitted the synthesis and comprehension of coherent text in an open-ended way, therefore translating the theoretical algorithms into practical applications. The large language models (LLMs) have significantly impacted businesses such as report summarization software and copywriters. Observations indicate, however, that LLMs may exhibit social prejudice and toxicity, posing ethical and societal dangers of consequences resulting from irresponsibility. Large-scale benchmarks for accountable LLMs should consequently be developed. Although several empirical investigations reveal the existence of a few ethical difficulties in advanced LLMs, there is little systematic examination and user study of the risks and harmful behaviors of current LLM usage. To further educate future efforts on constructing ethical LLMs responsibly, we perform a qualitative research method called ``red teaming'' on OpenAI's ChatGPT\footnote{In this paper, ChatGPT refers to the version released on Dec 15th.} to better understand the practical features of ethical dangers in recent LLMs. We analyze ChatGPT comprehensively from four perspectives: 1) extit{Bias} 2) extit{Reliability} 3) extit{Robustness} 4) extit{Toxicity}. In accordance with our stated viewpoints, we empirically benchmark ChatGPT on multiple sample datasets. We find that a significant number of ethical risks cannot be addressed by existing benchmarks, and hence illustrate them via additional case studies. In addition, we examine the implications of our findings on AI ethics and harmal behaviors of ChatGPT, as well as future problems and practical design considerations for responsible LLMs. We believe that our findings may give light on future efforts to determine and mitigate the ethical hazards posed by machines in LLM applications.

Motivation & Objective

• Characterize ethical risks (bias, toxicity) in ChatGPT using established benchmarks and human evaluation.
• Assess robustness and reliability of ChatGPT under perturbations, adversarial prompts, and factual questions.
• Identify real-world vulnerabilities (e.g., prompt injections) and language/cross-language biases across scenarios.
• Provide design considerations and guidance for responsible development of large language models.

Proposed method

• Red-team ChatGPT following HELM-aligned principles to assess four perspectives: bias, reliability, robustness, and toxicity.
• Benchmark ChatGPT on BBQ (bias in QA) and BOLD (text generation bias and stereotypes) with EM and bias-related metrics.
• Compare ChatGPT to InstructGPT (davinci-v2) and GPT-3 (davinci-v1) in zero-shot settings.
• Conduct case studies on language translation, code generation, and open-ended dialogue to reveal additional biases beyond benchmarks.
• Evaluate robustness via semantic perturbations (misspelling, formatting) and Contrast Sets on IMDB and BoolQ, using EM as the metric.
• Test adversarial prompt injection scenarios to assess safety and jailbreak susceptibility.
• Assess reliability through OpenBookQA and TruthfulQA and compare Exact Match results across models.

Figure 1 : Framework of diagnosing AI ethics of ChatGPT, with the comparisons of SOTA LLMs. The diagnosis focuses on four perspectives, 1) Bias , 2) Robustness , 3) Reliability and 4) Toxicity . The evaluation of each perspective consists of two parts, existing benchmarks and human-evaluated case st

Experimental results

Research questions

• RQ1 What are the ethical risks (bias, reliability, robustness, toxicity) exhibited by ChatGPT in practical use?
• RQ2 How do existing benchmarks capture or fail to capture these ethical risks in ChatGPT?
• RQ3 How vulnerable is ChatGPT to adversarial prompts and jailbreak attempts, and to what extent are safety mechanisms bypassed?
• RQ4 How does ChatGPT perform on bias and fairness across multilingual and code-generation contexts compared to other SOTA LLMs?
• RQ5 What design considerations emerge for mitigating harms and improving responsible deployment of LLMs?

Key findings

• ChatGPT shows lower bias levels than the baselines InstructGPT and GPT-3 on BBQ and BOLD benchmarks in zero-shot evaluation.
• ChatGPT achieves strong robustness on benchmarked tasks and outperforms baselines on semantic robustness in IMDB and BoolQ under perturbations.
• Several ethical risks beyond benchmarks are observed through case studies, including language understanding biases across languages and biased code-generation tendencies.
• ChatGPT demonstrates high safety in a controlled jailbreak prompt-injection study, with only 2 of 100 scenarios yielding direct responses despite many prompts bypassing safeguards.
• Adversarial prompt injections reveal that jailbreak attempts can bypass safety mechanisms in many cases, highlighting vulnerability in real-world use.
• Reliability results show ChatGPT maintains competitive exact-match performance on OpenBookQA and TruthfulQA relative to InstructGPT and GPT-3, with notable weaknesses in some knowledge domains.