[Paper Review] Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models

Chaoqun Liu, Wenxuan Zhang+3 more|arXiv (Cornell University)|Mar 15, 2024

Natural Language Processing Techniques4 citations

TL;DR

This study investigates whether translating multilingual queries into English is optimal for large language models (LLMs), finding that while translation improves performance on English-centric LLMs for non-English NLP tasks, native-language prompting outperforms translation for culture-intensive tasks. The key contribution is advocating for multilingual LLMs with equal proficiency across all languages, rather than relying on translation as a universal solution.

ABSTRACT

Large language models (LLMs) have demonstrated multilingual capabilities, yet they are mostly English-centric due to the imbalanced training corpora. While prior works have leveraged this bias to enhance multilingual performance through translation, they have been largely limited to natural language processing (NLP) tasks. In this work, we extend the evaluation to real-world user queries and non-English-centric LLMs, offering a broader examination of multilingual performance. Our key contribution lies in demonstrating that while translation into English can boost the performance of English-centric LLMs on NLP tasks, it is not universally optimal. For culture-related tasks that need deep language understanding, prompting in the native language proves more effective as it better captures the nuances of culture and language. Our experiments expose varied behaviors across LLMs and tasks in the multilingual context, underscoring the need for a more comprehensive approach to multilingual evaluation. Therefore, we call for greater efforts in developing and evaluating LLMs that go beyond English-centric paradigms.

Motivation & Objective

• To evaluate the effectiveness of translating multilingual queries into English for improving LLM performance across diverse languages.
• To compare translation-based prompting with native-language and cross-lingual prompting strategies in real-world and NLP benchmark scenarios.
• To investigate whether English-centric LLMs benefit from translation or if native-language prompting yields superior results for culturally nuanced tasks.
• To assess the performance of multilingual LLMs optimized for non-English languages in tasks requiring deep cultural understanding.
• To advocate for the development of truly multilingual LLMs with balanced capabilities across all languages, rather than relying on translation as a workaround.

Proposed method

• Conducting controlled experiments across multiple LLMs (e.g., Llama-2, Mistral, Llama-70B, ChatGPT) and translation tools (Google Translate, NLLB, XLT) to compare prompting strategies.
• Evaluating prompting strategies: native-language basic, native CoT, English basic, English CoT, XLT (cross-lingual transfer), and translation via Google/NLLB.
• Using standardized benchmarks: MKQA (multilingual question answering), M3Exam (multilingual reasoning), and XL-sum (multilingual summarization) for NLP tasks.
• Testing real-world multilingual user queries from ShareGPT to assess practical performance in open-ended, non-structured scenarios.
• Employing GPT-4-Turbo as a judge model to evaluate response quality using a LLM-as-a-judge prompt for consistency and reliability.
• Designing a prompt to detect whether a request requires local cultural knowledge, enabling targeted analysis of culture-sensitive tasks.

Experimental results

Research questions

• RQ1 Does translating multilingual queries into English consistently improve performance across diverse NLP tasks for English-centric LLMs?
• RQ2 How does native-language prompting compare to translation-based prompting in real-world multilingual user queries?
• RQ3 Under what conditions does translation degrade performance, particularly in tasks requiring cultural or linguistic nuance?
• RQ4 Can multilingual LLMs optimized for non-English languages outperform English-centric models when prompted in their native language?
• RQ5 Is translation a sufficient or optimal strategy for achieving robust multilingual performance, or is there a need for truly multilingual models?

Key findings

• For low-resource languages, translation into English significantly improves performance, with Trans-Google achieving 47.9 F1 on MKQA for Llama-2-70B-Chat in the Turkish language.
• On culture-related tasks, prompting in the native language yields superior results—e.g., Llama-2-70B-Chat achieved 39.7 ROUGE-1 on XL-sum for Chinese using native prompting, outperforming translation-based methods.
• For high-resource languages like Chinese and French, native-language prompting with CoT (Chain-of-Thought) consistently outperformed translation, with Llama-2-70B-Chat achieving 34.5 ROUGE-1 for French using native CoT.
• Translation-based prompting (e.g., Trans-Google) improved performance on language-agnostic tasks (e.g., MGSM math), but degraded performance on culture-dependent tasks, such as a Chinese question requiring local context.
• Multilingual LLMs optimized for non-English languages showed stronger native-language performance, indicating that language-specific fine-tuning enhances cultural nuance understanding.
• The study concludes that translation is not universally optimal and that developing truly multilingual LLMs is essential to overcome the limitations of English-centric models.