[論文レビュー] On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective

Jindong Wang, Xixu Hu+11 more|arXiv (Cornell University)|Feb 22, 2023

Adversarial Robustness in Machine Learning被引用数 90

ひとこと要約

本論文は、ChatGPT の adversarial inputs および out-of-distribution データに対するゼロショット設定での頑健性を評価し、複数の NLP タスクにおいて他のファウンデーションモデルと比較し、制約と今後の方向性を論じる。

ABSTRACT

ChatGPT is a recent chatbot service released by OpenAI and is receiving increasing attention over the past few months. While evaluations of various aspects of ChatGPT have been done, its robustness, i.e., the performance to unexpected inputs, is still unclear to the public. Robustness is of particular concern in responsible AI, especially for safety-critical applications. In this paper, we conduct a thorough evaluation of the robustness of ChatGPT from the adversarial and out-of-distribution (OOD) perspective. To do so, we employ the AdvGLUE and ANLI benchmarks to assess adversarial robustness and the Flipkart review and DDXPlus medical diagnosis datasets for OOD evaluation. We select several popular foundation models as baselines. Results show that ChatGPT shows consistent advantages on most adversarial and OOD classification and translation tasks. However, the absolute performance is far from perfection, which suggests that adversarial and OOD robustness remains a significant threat to foundation models. Moreover, ChatGPT shows astounding performance in understanding dialogue-related texts and we find that it tends to provide informal suggestions for medical tasks instead of definitive answers. Finally, we present in-depth discussions of possible research directions.

研究の動機と目的

• ChatGPT’s adversarial robustness on standard NLP benchmarks (AdvGLUE, ANLI) and an adversarial translation task.
• Evaluate ChatGPT’s out-of-distribution robustness on new datasets (Flipkart, DDXPlus) in a zero-shot setting.
• Compare ChatGPT’s performance to a range of large foundation models under adversarial and OOD conditions.
• Provide analysis and discussion of robustness challenges and potential research directions for foundation models.

提案手法

• Use zero-shot evaluation on AdvGLUE and ANLI to assess adversarial robustness via attack success rate (ASR).
• Evaluate OOD robustness on Flipkart and DDXPlus using F1-score as the metric.
• Include zero-shot machine translation evaluation on AdvGLUE-T comparing ChatGPT with fine-tuned MT models and GPT-family baselines (BLEU, GLEU, METEOR).
• Select a set of representative foundation models from HuggingFace and the OpenAI API as baselines for comparison.
• Promote prompts-based evaluation and manual processing of outputs to ensure comparability across models.

Figure 1: Robustness evaluation of different foundation models: performance vs. parameter size. Results show that ChatGPT shows consistent advantage on adversarial and OOD classification tasks. However, its absolute performance is far from perfection, indicating much room for improvement.

実験結果

リサーチクエスチョン

• RQ1 How robust is ChatGPT to adversarial perturbations in text classification and NLI tasks in a zero-shot setting?
• RQ2 How does ChatGPT perform on out-of-distribution (OOD) data compared to other large foundation models?
• RQ3 What are the relative strengths and weaknesses of ChatGPT in translation and dialogue-related NLP tasks under robustness challenges?
• RQ4 What implications do adversarial and OOD robustness have for the deployment of ChatGPT in safety-critical or domain-shifted applications?

主な発見

• ChatGPT shows consistent improvements over many baselines on adversarial classification tasks, but absolute performance remains imperfect.
• ChatGPT and other GPT-family models perform well on OOD datasets, with notable strength on DDXPlus (medical dialogue) relative to many competitors.
• ChatGPT demonstrates strong translation readability under adversarial inputs, though translation performance can lag behind some instruction-tuned peers in certain metrics.
• Medical-related responses from ChatGPT tend to offer informed analysis and recommendations rather than definitive diagnoses, highlighting safety-conscious behavior.
• Larger instruction-tuned models (e.g., Flan-T5-L) can approach or match performance of larger models on some tasks, suggesting benefits of instruction tuning for robustness.
• The study emphasizes that zero-shot robustness of many foundation models remains a vulnerability, underscoring the need for robust training and defense strategies.