[Paper Review] MasakhaNEWS: News Topic Classification for African languages

David Ifeoluwa Adelani, Marek Masiak+63 more|arXiv (Cornell University)|Apr 19, 2023

Topic Modeling4 citations

TL;DR

This paper introduces MasakhaNEWS, the largest news topic classification dataset for 16 African languages, enabling benchmarking of NLP models across typologically diverse languages. Using few-shot learning with Pattern-Exploiting Training (PET), the authors achieve 86.0 F1 score with just 10 labeled examples per class—over 90% of the performance of fully supervised fine-tuning (92.6 F1), demonstrating strong transferability of existing NLP techniques to low-resource African languages.

ABSTRACT

African languages are severely under-represented in NLP research due to lack of datasets covering several NLP tasks. While there are individual language specific datasets that are being expanded to different tasks, only a handful of NLP tasks (e.g. named entity recognition and machine translation) have standardized benchmark datasets covering several geographical and typologically-diverse African languages. In this paper, we develop MasakhaNEWS -- a new benchmark dataset for news topic classification covering 16 languages widely spoken in Africa. We provide an evaluation of baseline models by training classical machine learning models and fine-tuning several language models. Furthermore, we explore several alternatives to full fine-tuning of language models that are better suited for zero-shot and few-shot learning such as cross-lingual parameter-efficient fine-tuning (like MAD-X), pattern exploiting training (PET), prompting language models (like ChatGPT), and prompt-free sentence transformer fine-tuning (SetFit and Cohere Embedding API). Our evaluation in zero-shot setting shows the potential of prompting ChatGPT for news topic classification in low-resource African languages, achieving an average performance of 70 F1 points without leveraging additional supervision like MAD-X. In few-shot setting, we show that with as little as 10 examples per label, we achieved more than 90\% (i.e. 86.0 F1 points) of the performance of full supervised training (92.6 F1 points) leveraging the PET approach.

Motivation & Objective

• Address the lack of benchmark datasets for news topic classification in low-resource African languages.
• Enable evaluation of multilingual language models (LLMs) across diverse African languages, including non-Latin scripts.
• Investigate few-shot and zero-shot learning techniques for effective adaptation of LLMs to low-resource African languages.
• Provide baseline models using classical ML and fine-tuned LLMs to establish performance benchmarks.
• Explore parameter-efficient and prompt-based fine-tuning methods to reduce data and compute demands for African language NLP.

Proposed method

• Constructed MasakhaNEWS, a multilingual news topic classification dataset with 16 African languages, using publicly available news articles and human-annotated labels.
• Evaluated classical machine learning models (e.g., SVM, logistic regression) and fine-tuned multilingual LLMs (e.g., mBERT, XLM-R) as baseline models.
• Applied cross-lingual parameter-efficient fine-tuning via MAD-X to adapt models with minimal parameters.
• Utilized Pattern-Exploiting Training (PET) to leverage verbalizer templates and few-shot labeled data without full fine-tuning.
• Explored prompting strategies using GPT-3.5 and GPT-4 via API for zero-shot and few-shot inference.
• Evaluated prompt-free fine-tuning using SetFit and the co:here embedding API to compare performance without prompting.

Figure 1: Comparison of article content type used for training news topic classification models . We report the average across all languages when either headline or headline+text is used

Experimental results

Research questions

• RQ1 Can few-shot learning approaches like PET achieve high performance on African language news topic classification with minimal labeled data?
• RQ2 How do prompt-based methods (e.g., GPT-3.5/4) compare to fine-tuned and parameter-efficient methods on African languages, especially those with non-Latin scripts?
• RQ3 To what extent can cross-lingual transfer improve performance when fine-tuning on low-resource African languages using data from high-resource languages?
• RQ4 Do existing supervised NLP techniques generalize effectively to African languages across diverse linguistic families and writing systems?
• RQ5 What is the performance ceiling of prompt-free fine-tuning (e.g., SetFit, co:here) in few-shot settings for African language text classification?

Key findings

• With only 10 labeled examples per class, the PET method achieved an F1 score of 86.0, representing over 90% of the performance of full-supervised fine-tuning (92.6 F1).
• PET outperformed standard fine-tuning baselines in few-shot settings, achieving 89.9 F1 with 50 shots—close to the full-supervised baseline.
• Prompting GPT-3.5 and GPT-4 performed poorly on African languages, especially those with non-Latin scripts, indicating limited zero-shot generalization for this task.
• Cross-lingual transfer from Swahili to Luganda yielded a +9.0 F1 improvement, likely due to their shared Bantu language subgroup affiliation.
• SetFit and the co:here API achieved competitive results without prompting, with SetFit matching PET’s 50-shot performance (89.9 F1) without using unlabeled data.
• The study confirms that existing NLP techniques, including fine-tuning and parameter-efficient adaptation, are viable and effective for African languages with minimal labeled data.

Figure 2: Interface of our in-house Annotation tool . Annotators can correct the pre-defined category assigned and also edit their annotation