[Paper Review] Multilingual Tourist Assistance using ChatGPT: Comparing Capabilities in Hindi, Telugu, and Kannada

Sanjana Kolar, Rohit Kumar|arXiv (Cornell University)|Jul 28, 2023

Artificial Intelligence in Healthcare and EducationMedicine8 citations

TL;DR

The paper evaluates ChatGPT's English-to-Indian-language translations (Hindi, Kannada, Telugu) using a 50-question BLEU-based and human-evaluated study, finding Hindi superior overall.

ABSTRACT

This research investigates the effectiveness of ChatGPT, an AI language model by OpenAI, in translating English into Hindi, Telugu, and Kannada languages, aimed at assisting tourists in India's linguistically diverse environment. To measure the translation quality, a test set of 50 questions from diverse fields such as general knowledge, food, and travel was used. These were assessed by five volunteers for accuracy and fluency, and the scores were subsequently converted into a BLEU score. The BLEU score evaluates the closeness of a machine-generated translation to a human translation, with a higher score indicating better translation quality. The Hindi translations outperformed others, showcasing superior accuracy and fluency, whereas Telugu translations lagged behind. Human evaluators rated both the accuracy and fluency of translations, offering a comprehensive perspective on the language model's performance.

Motivation & Objective

• Assess how well ChatGPT translates English into Hindi, Kannada, and Telugu for tourist information in India.
• Quantify translation quality using subjective (accuracy and fluency ratings) and objective (BLEU scores) measures.
• Identify language-specific strengths and weaknesses to guide improvements for tourist-domain translations.

Proposed method

• Use gpt-3.5-turbo to translate English text to target Indian languages via a two-part prompt (systemRole + userRole).
• Evaluate translations with 5 native-language volunteers on accuracy and fluency (scale 1-5).
• Compute BLEU scores (0-100) by comparing machine translations to reference human translations for 50 questions.
• Categorize 60 initial questions into 3 themes (General, Food, Travel) and select 50 relevant items.

Experimental results

Research questions

• RQ1 Can ChatGPT accurately translate English tourist queries into Hindi, Kannada, and Telugu for German/International visitors?
• RQ2 How do subjective (accuracy/fluency) and objective (BLEU) evaluations compare across the three languages?
• RQ3 What language-specific improvements are suggested to enhance tourist-domain translations?

Key findings

• Hindi translations show highest accuracy and fluency overall (e.g., General: 4.8 accuracy, 4.6 fluency).
• Telugu translations have the lowest performance (BLEU: 13.12; General accuracy: 2.6; fluency: 2.1).
• Kannada translations are intermediate (BLEU: 46.78; General accuracy: 3.7; fluency: 3.5).
• Overall Hindi general/theme translations score higher than Kannada and Telugu on accuracy and fluency.