[Paper Review] A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers

Kaiyu Huang, Fengran Mo+11 more|arXiv (Cornell University)|May 17, 2024

Topic Modeling5 citations

TL;DR

A comprehensive survey of LLMs with multilingual capabilities, presenting a taxonomy, training and inference strategies, security and multi-domain considerations, datasets, benchmarks, and future directions.

ABSTRACT

The rapid development of Large Language Models (LLMs) demonstrates remarkable multilingual capabilities in natural language processing, attracting global attention in both academia and industry. To mitigate potential discrimination and enhance the overall usability and accessibility for diverse language user groups, it is important for the development of language-fair technology. Despite the breakthroughs of LLMs, the investigation into the multilingual scenario remains insufficient, where a comprehensive survey to summarize recent approaches, developments, limitations, and potential solutions is desirable. To this end, we provide a survey with multiple perspectives on the utilization of LLMs in the multilingual scenario. We first rethink the transitions between previous and current research on pre-trained language models. Then we introduce several perspectives on the multilingualism of LLMs, including training and inference methods, information retrieval, model security, multi-domain with language culture, and usage of datasets. We also discuss the major challenges that arise in these aspects, along with possible solutions. Besides, we highlight future research directions that aim at further enhancing LLMs with multilingualism. The survey aims to help the research community address multilingual problems and provide a comprehensive understanding of the core concepts, key techniques, and latest developments in multilingual natural language processing based on LLMs.

Motivation & Objective

• Define and clarify the concept of LLMs with multilingualism and motivate the need for language-fair, accessible AI.
• Provide a structured taxonomy and comparative analysis of multilingual LLMs across training, inference, security, and multi-domain applications.
• Identify challenges and propose potential solutions to improve multilingual capabilities and accessibility of LLMs.
• Summarize datasets, benchmarks, and evaluation protocols relevant to multilingual LLMs and outline future research directions.

Proposed method

• Propose a structured taxonomy to categorize multilingual LLMs and clarify paradigm shifts from pre-training to prompt/predict in multilingual settings.
• Survey representative models trained from scratch and via continual training, detailing architectures, data, and training strategies.
• Analyze multilingual inference strategies (direct vs pre-translation) and report empirical performance on multilingual tasks.
• Discuss security, open-source vs closed-source models, and multi-domain adaptations (medical, legal, etc.).
• Curate and summarize datasets and benchmarks used for multilingual evaluation and highlight cross-lingual bias considerations.

Experimental results

Research questions

• RQ1 What are the key factors and architectural choices enabling LLMs to excel in multilingual settings?
• RQ2 How do training from scratch versus continual training impact multilingual abilities across languages and domains?
• RQ3 How do inference strategies (direct multilingual inference vs pre-translation) compare in real multilingual tasks?
• RQ4 What are the major safety, security, and bias concerns for multilingual LLMs, and how can they be mitigated?
• RQ5 What datasets and benchmarks best capture multilingual capabilities and cross-lingual performance?

Key findings

Model	Inference	XCOPA Acc	XStoryCloze Acc	BELEBELE Acc	XLSum Rouge-L	XQuAD F1	TyDiQA-GP F1
PaLM2-S	En-Pivot	87.3	96.4	76.7	23.7	67.2	81.6
PaLM2-S	Direct	89.7	96.8	77.8	26.8	70.7	83.8
PaLM2-L	En-Pivot	89.6	97.8	84.3	25.4	78.7	81.0
PaLM2-L	Direct	93.4	99.1	88.4	28.0	85.9	83.0

• Multilingual LLMs now support hundreds of languages with improved non-English performance for direct inference.
• Direct multilingual inference often outperforms pre-translation approaches on several tasks, reducing translation errors and preserving linguistic nuances.
• Training from scratch and continual training both enhance multilingual abilities, with continual training enabling targeted language expansion while leveraging existing knowledge.
• Security, model accessibility (open vs closed), and cross-domain adaptability are critical considerations for practical multilingual deployment.
• A broad set of multilingual corpora and benchmarks (e.g., CROSS-LINGUAL and cross-domain datasets) are essential for evaluating and guiding multilingual capabilities.