[论文解读] ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning

Viet Dac Lai, Nghia Trung Ngo+5 more|arXiv (Cornell University)|Apr 12, 2023

Topic Modeling被引用 51

一句话总结

本论文在 7 个 NLP 任务和 37 种语言上对 ChatGPT 进行了大规模的零-shot 评估，以评估多语种性能，结果表明 ChatGPT 通常不如监督模型，并显示出英语偏向，尤其在复杂任务上。

ABSTRACT

Over the last few years, large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) that fundamentally transform research and developments in the field. ChatGPT represents one of the most exciting LLM systems developed recently to showcase impressive skills for language generation and highly attract public attention. Among various exciting applications discovered for ChatGPT in English, the model can process and generate texts for multiple languages due to its multilingual training data. Given the broad adoption of ChatGPT for English in different problems and areas, a natural question is whether ChatGPT can also be applied effectively for other languages or it is necessary to develop more language-specific technologies. The answer to this question requires a thorough evaluation of ChatGPT over multiple tasks with diverse languages and large datasets (i.e., beyond reported anecdotes), which is still missing or limited in current research. Our work aims to fill this gap for the evaluation of ChatGPT and similar LLMs to provide more comprehensive information for multilingual NLP applications. While this work will be an ongoing effort to include additional experiments in the future, our current paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. We also focus on the zero-shot learning setting for ChatGPT to improve reproducibility and better simulate the interactions of general users. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages, calling for further research to develop better models and understanding for multilingual learning.

研究动机与目标

• 在超越英语的多样化 NLP 任务上评估 ChatGPT 及类似大语言模型，以了解多语种能力与局限。
• 覆盖高资源、中等资源、低资源和极低资源语言，以反映现实世界的多语言环境。
• 优先采用零-shot 学习以模拟一般用户互动并最大程度提高可重复性。
• 提供与最先进的监督模型的对比基线以供参照。

提出的方法

• 在七个 NLP 任务上评估 ChatGPT：词性标注（POS tagging）、命名实体识别（NER）、关系抽取、自然语言推理、问答、常识推理，以及摘要。
• 覆盖 37 种语言，涵盖高资源到极低资源类别，使用零-shot 提示。
• 使用英文任务描述和语言特定的翻译来提示 ChatGPT，并用任务特定规则解析输出。
• 与强基线监督模型进行比较（例如 POS 的 XLM-R、NER 的 DAMO、RE 的 mT5-IL）以对结果进行上下文分析。
• 将零-shot 结果作为对普通用户性能的主要证据并讨论语言相关偏差。

Figure 7: Input prompt and output of ChatGPT for Wikipedia Cloze QA dataset (IndicNLPSuite). Translation of the statement and options by Google Translate: Ratan Devasi was born on 25 September 1975 at Mount Abu in the Sirohi district of $<$ MASK $>$ . His father’s name is Shankarlal Devasi and wife’

实验结果

研究问题

• RQ1 ChatGPT 在广泛的语言集上对多种 NLP 任务的零-shot 表现如何？
• RQ2 ChatGPT 是否存在英语语言偏见，以及这种偏见在高资源到极低资源语言中的表现如何？
• RQ3 ChatGPT 的多语言表现与每个任务的最先进监督模型相比如何？
• RQ4 在非英语 NLP 应用中使用多语言大型语言模型的实际意义和影响是什么？

主要发现

• 在大多数研究任务和语言中，ChatGPT 的零-shot 表现通常不如最先进的监督模型。
• ChatGPT 在英语上的表现优于其他语言，尤其是在需要复杂推理的任务上。
• 在许多语言的若干任务中，英语提示往往产生与语言特定提示相同或更好的结果。
• ChatGPT 的多语言评估涵盖 7 个任务和 37 种语言，凸显显著差距以及对语言特定模型或改进的需求。
• 研究表明存在对英语的偏见，并呼吁进一步研究以改进 LLM 的多语言学习。