[Paper Review] Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine

Wenxiang Jiao, Wenxuan Wang+4 more|arXiv (Cornell University)|Jan 20, 2023

Artificial Intelligence in Healthcare and EducationMedicine313 citations

TL;DR

The paper preliminarily evaluates ChatGPT for machine translation, showing GPT-4 markedly boosts performance to rival commercial systems, with pivot prompting helping distant languages.

ABSTRACT

This report provides a preliminary evaluation of ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. We adopt the prompts advised by ChatGPT to trigger its translation ability and find that the candidate prompts generally work well with minor performance differences. By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but exhibits good results on spoken language. Further, we explore an interesting strategy named $\mathbf{pivot~prompting}$ for distant languages, which asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, improving the translation performance noticeably. With the launch of the GPT-4 engine, the translation performance of ChatGPT is significantly boosted, becoming comparable to commercial translation products, even for distant languages. Human analysis on Google Translate and ChatGPT suggests that ChatGPT with GPT-3.5 tends to generate more hallucinations and mis-translation errors while that with GPT-4 makes the least errors. In other words, ChatGPT has already become a good translator. Please refer to our Github project for more details: https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator

Motivation & Objective

• Assess how translation prompts affect ChatGPT output.
• Evaluate multilingual translation performance across high-resource and distant languages.
• Investigate translation robustness on domain-specific and noisy data.
• Explore pivot prompting to improve translation quality for distant language pairs.
• Assess impact of GPT-4 as the translation engine on ChatGPT’s performance.

Proposed method

• Use three candidate prompts (Tp1, Tp2, Tp3) to trigger translation and compare it with Google Translate, DeepL, and Tencent TranSmart.
• Evaluate on Flores-101 for multilingual translation across 101 languages.
• Test robustness on WMT19 Bio and WMT20 Rob2/Rob3 robustness sets.
• Compare direct ChatGPT translations with a pivot prompting approach (source to pivot to target).
• Re-evaluate translations using GPT-4 to measure engine impact.

Experimental results

Research questions

• RQ1 How do translation prompts influence ChatGPT’s translation quality across language pairs?
• RQ2 How does ChatGPT perform relative to commercial systems on high-resource vs. distant languages?
• RQ3 Do pivot prompting strategies improve translation quality for distant language pairs?
• RQ4 What is the impact of GPT-4 as the engine on ChatGPT’s translation performance?
• RQ5 How robust are ChatGPT translations to domain-specific or noisy text?

Key findings

• ChatGPT with Tp3 generally translates reasonably well but lags commercial baselines on Chinese→English; Tp3 is the best among the candidate prompts.
• ChatGPT is competitive with Google Translate and DeepL for German↔English but much weaker for Romanian↔English, highlighting resource and language-family effects.
• Pivot prompting significantly improves translation for distant languages (e.g., De/Zh, Ro/Zh) by up to ~6.6 BLEU points in reported cases.
• GPT-4 dramatically boosts ChatGPT translation performance, achieving levels comparable to top commercial systems in zero-shot settings across several directions.
• Human analysis shows GPT-4 has the fewest translation errors among the compared systems, though GPT-4 BLEU remains below Google Translate in some cases.