[Paper Review] All Languages Matter: On the Multilingual Safety of Large Language Models

Wenxuan Wang, Zhaopeng Tu+5 more|arXiv (Cornell University)|Oct 2, 2023

Topic Modeling4 citations

TL;DR

This paper introduces XSafety, the first multilingual safety benchmark evaluating 14 safety risks across 10 languages, and demonstrates that large language models (LLMs) produce significantly more unsafe responses in non-English languages than in English. To address this, the authors propose cross-lingual prompting methods—such as instructing models to reason in English before responding in the source language—that reduce unsafe responses in non-English queries from 19.1% to 9.7% for ChatGPT.

ABSTRACT

Safety lies at the core of developing and deploying large language models (LLMs). However, previous safety benchmarks only concern the safety in one language, e.g. the majority language in the pretraining data such as English. In this work, we build the first multilingual safety benchmark for LLMs, XSafety, in response to the global deployment of LLMs in practice. XSafety covers 14 kinds of commonly used safety issues across 10 languages that span several language families. We utilize XSafety to empirically study the multilingual safety for 4 widely-used LLMs, including both close-API and open-source models. Experimental results show that all LLMs produce significantly more unsafe responses for non-English queries than English ones, indicating the necessity of developing safety alignment for non-English languages. In addition, we propose several simple and effective prompting methods to improve the multilingual safety of ChatGPT by evoking safety knowledge and improving cross-lingual generalization of safety alignment. Our prompting method can significantly reduce the ratio of unsafe responses from 19.1% to 9.7% for non-English queries. We release our data at https://github.com/Jarviswang94/Multilingual_safety_benchmark.

Motivation & Objective

• To address the lack of multilingual safety evaluation in LLMs, especially for non-English languages.
• To investigate whether safety alignment trained primarily on English data generalizes effectively to other languages.
• To develop and evaluate prompting strategies that improve multilingual safety without fine-tuning.
• To provide a comprehensive, multilingual safety benchmark to support future research on cross-lingual model safety.

Proposed method

• Constructed XSafety, a multilingual safety benchmark with 28,000 annotated instances across 10 languages and 14 safety categories.
• Translated existing monolingual safety benchmarks into 10 languages using professional translators to ensure linguistic and cultural accuracy.
• Evaluated four widely used LLMs—ChatGPT, PaLM 2, LLaMA2-Chat, and Vicuna—on XSafety to assess multilingual safety performance.
• Proposed three prompting strategies: SafePrompt (safety knowledge evocation), XLingPrompt (cross-lingual reasoning in English), and XSafePrompt (hybrid prompting).
• Used zero-shot prompting to test the effectiveness of these methods on non-English queries without model fine-tuning.
• Compared model outputs across prompting methods using both automatic metrics and human evaluation for safety classification.

Experimental results

Research questions

• RQ1 How does the safety performance of LLMs vary across different languages, particularly between English and non-English languages?
• RQ2 To what extent does English-focused safety alignment generalize to non-English languages in LLMs?
• RQ3 Can prompting techniques improve the multilingual safety of LLMs without fine-tuning?
• RQ4 Which prompting strategy—safety knowledge evocation or cross-lingual reasoning—yields the most effective safety generalization across languages?
• RQ5 What are the linguistic and structural factors that affect the transferability of safety alignment across languages?

Key findings

• All evaluated LLMs produce significantly more unsafe responses for non-English queries than for English queries, with unsafe response rates rising from ~19.1% in non-English to ~10% in English.
• The most effective prompting method, XSafePrompt, reduced unsafe responses in non-English queries from 19.1% to 9.7%, representing a 50% relative reduction.
• XLingPrompt1, which instructs models to reason in English before responding in the source language, was especially effective for Russian, highlighting cross-lingual generalization potential.
• The performance of prompting strategies varied significantly across languages, with higher translation difficulty correlating with lower effectiveness, indicating linguistic diversity affects safety generalization.
• Combining SafePrompt and XLingPrompt did not improve performance, suggesting conflicting mechanisms between knowledge evocation and cross-lingual reasoning.
• Responses from XLingPrompt1 closely mirrored those in English, confirming that cross-lingual reasoning improves the transfer of safety alignment from English to other languages.