[Paper Review] Phoenix: Democratizing ChatGPT across Languages

Zhihong Chen, Feng Jiang+12 more|arXiv (Cornell University)|Apr 20, 2023

Artificial Intelligence in Healthcare and Education20 citations

TL;DR

Phoenix is an open-source multilingual LLM based on BLOOMZ (Phoenix) and a Latin version Chimera (LLaMA backbone) that democratizes ChatGPT-like capabilities across many languages, achieving strong performance in non-Latin languages and competitive results with English models.

ABSTRACT

This paper presents our efforts to democratize ChatGPT across language. We release a large language model "Phoenix", achieving competitive performance among open-source English and Chinese models while excelling in languages with limited resources (covering both Latin and non-Latin languages). We believe this work will be beneficial to make ChatGPT more accessible, especially in countries where people cannot use ChatGPT due to restrictions from OpenAI or local goverments. Our data, code, and models are available at https://github.com/FreedomIntelligence/LLMZoo.

Motivation & Objective

• Lower barriers to building and researching ChatGPT-like dialogue models in diverse languages.
• Leverage both instruction-following and conversation data to train a multilingual LLM.
• Assess cross-language performance and establish benchmarks against existing open-source and closed-source models.
• Demonstrate the value of multilingual data in pre-training and post-training stages to reduce language-specific biases.

Proposed method

• Construct multilingual instruction data by combining Alpaca-derived data, post-translation, and user-centered instruction datasets translated into 40 languages.
• Gather conversation data from ShareGPT and Discord channels, then translate into 40 languages to support multilingual chat training.
• Fine-tune a BLOOMZ-based backbone to form Phoenix (multilingual) and a Latin-chassis Chimera (LLaMA backbone) by integrating instruction-following and multi-turn conversations.
• Train with max context length 2048, using AdamW, batch size 256, 3 epochs, learning rate 2e-5, and no weight decay.
• Evaluate using automatic (GPT-4 and GPT-3.5 Turbo as reviewers) and human assessments across Chinese, English, and multiple non-Latin languages.
• Compare against open-source LLMs (e.g., BELLE, Chinese-Alpaca, Vicuna) and selective closed models (ChatGPT, Baidu-Wenxin) to position Phoenix.

Experimental results

Research questions

• RQ1 Can a multilingual LLM trained with both instruction-following and conversation data surpass existing open-source models across multiple languages?
• RQ2 How does the multilingual tax impact performance in non-Latin languages, and can this be mitigated by backbone choice (BLOOMZ vs. LLaMA)?
• RQ3 What is the relative performance of Phoenix and Chimera across Latin and non-Latin languages compared to state-of-the-art open-source and closed-source models?
• RQ4 Does integrating instruction data with conversation data yield measurable gains over using either data type alone?

Key findings

• Phoenix achieves state-of-the-art performance among open-source Chinese LLMs and outperforms many non-Latin language models.
• In non-Latin languages, Phoenix largely outperforms existing open-source LLMs, including Arabic, Japanese, and Korean benchmarks.
• Chimera (Latin Phoenix) attains GPT-4-like quality with 96.6% ChatGPT Quality on GPT-4 assessment, setting a new open-source SOTA.
• Phoenix shows strong competitiveness with Baidu-Wenxin and BELLE in Chinese and English benchmarks, though some non-open-source models remain higher in English.
• Ablation results indicate that adding instruction data yields ~5-6% relative improvement for Phoenix/Chimera in target tasks.
• Human evaluation indicates Phoenix often dominates or matches several established open-source Chinese models and remains competitive with commercial models.

Figure 1 : Language Distribution in Our Dataset: Top 15 Languages Represented out of 133.