[Paper Review] BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLP

Mohsinul Kabir, Mohammed Saidul Islam+4 more|arXiv (Cornell University)|Sep 22, 2023

Topic Modeling4 citations

TL;DR

This paper introduces BenLLMEval, a comprehensive zero-shot evaluation of large language models (LLMs) on Bengali NLP tasks, including text summarization, question answering, and sentiment analysis. Despite strong performance on some tasks, open-source LLMs like LLaMA-2-13b-chat show significantly lower performance than state-of-the-art fine-tuned models, highlighting critical gaps in low-resource language capabilities and the need for targeted improvements in Bengali NLP.

ABSTRACT

Large Language Models (LLMs) have emerged as one of the most important breakthroughs in NLP for their impressive skills in language generation and other language-specific tasks. Though LLMs have been evaluated in various tasks, mostly in English, they have not yet undergone thorough evaluation in under-resourced languages such as Bengali (Bangla). To this end, this paper introduces BenLLM-Eval, which consists of a comprehensive evaluation of LLMs to benchmark their performance in the Bengali language that has modest resources. In this regard, we select various important and diverse Bengali NLP tasks, such as text summarization, question answering, paraphrasing, natural language inference, transliteration, text classification, and sentiment analysis for zero-shot evaluation of popular LLMs, namely, GPT-3.5, LLaMA-2-13b-chat, and Claude-2. Our experimental results demonstrate that while in some Bengali NLP tasks, zero-shot LLMs could achieve performance on par, or even better than current SOTA fine-tuned models; in most tasks, their performance is quite poor (with the performance of open-source LLMs like LLaMA-2-13b-chat being significantly bad) in comparison to the current SOTA results. Therefore, it calls for further efforts to develop a better understanding of LLMs in modest-resourced languages like Bengali.

Motivation & Objective

• To evaluate the zero-shot capabilities of large language models on under-resourced Bengali NLP tasks.
• To benchmark popular LLMs—GPT-3.5, LLaMA-2-13b-chat, and Claude-2—across diverse Bengali NLP tasks.
• To compare zero-shot LLM performance against state-of-the-art fine-tuned models in Bengali.
• To identify the strengths and limitations of current LLMs in low-resource language settings like Bengali.
• To provide a foundation for future research in improving LLM performance for low-resource languages.

Proposed method

• The study constructs a benchmark suite covering seven diverse Bengali NLP tasks: text summarization, question answering, paraphrasing, natural language inference, transliteration, text classification, and sentiment analysis.
• Zero-shot inference is applied to evaluate GPT-3.5, LLaMA-2-13b-chat, and Claude-2 without fine-tuning on Bengali data.
• Evaluation is conducted using standard metrics such as ROUGE for summarization, F1 for question answering, and accuracy for classification tasks.
• The benchmark is designed to reflect real-world diversity and challenge LLM generalization in low-resource settings.
• Performance is compared against existing state-of-the-art fine-tuned models to assess zero-shot transfer capability.

Experimental results

Research questions

• RQ1 How do major LLMs perform in zero-shot settings on diverse Bengali NLP tasks?
• RQ2 To what extent do zero-shot LLMs match or exceed the performance of fine-tuned state-of-the-art models in Bengali?
• RQ3 Why do open-source LLMs like LLaMA-2-13b-chat underperform on Bengali benchmarks despite strong English capabilities?
• RQ4 What are the key failure modes of LLMs when applied to low-resource languages like Bengali?
• RQ5 What insights can be drawn to guide future development of LLMs for under-resourced languages?

Key findings

• GPT-3.5 achieves competitive performance on certain Bengali NLP tasks, such as question answering and sentiment analysis, often matching or surpassing fine-tuned SOTA models.
• LLaMA-2-13b-chat exhibits significantly lower performance across most tasks, indicating poor zero-shot generalization in Bengali.
• Claude-2 shows strong zero-shot performance on several tasks, suggesting model architecture and pretraining data quality are critical factors.
• In tasks like text summarization and natural language inference, zero-shot LLMs consistently underperform compared to fine-tuned SOTA models.
• The study reveals a substantial performance gap between English and Bengali zero-shot NLP, underscoring the need for language-specific adaptation.
• Overall, while some LLMs show promise in zero-shot Bengali NLP, the results highlight significant limitations, especially for open-source models.