[Paper Review] ScandEval: A Benchmark for Scandinavian Natural Language Processing

Dan Saattrup Nielsen|arXiv (Cornell University)|Apr 3, 2023

Topic Modeling5 citations

TL;DR

This paper introduces ScandEval, a benchmarking framework for Scandinavian natural language processing that evaluates pretrained models across four tasks—named entity recognition, sentiment classification, linguistic acceptability, and question answering—using two newly released datasets: ScaLA and ScandiQA. The framework, implemented as a Python package and CLI, enables reproducible benchmarking of over 100 models on the Hugging Face Hub, with results hosted on an interactive online leaderboard. Key findings show substantial cross-lingual transfer among Mainland Scandinavian languages (Danish, Norwegian, Swedish), while transfer to Insular Scandinavian languages (Icelandic, Faroese) is limited, and national models outperform general multilingual models like XLM-RoBERTa and mDeBERTaV3.

ABSTRACT

This paper introduces a Scandinavian benchmarking platform, ScandEval, which can benchmark any pretrained model on four different tasks in the Scandinavian languages. The datasets used in two of the tasks, linguistic acceptability and question answering, are new. We develop and release a Python package and command-line interface, scandeval, which can benchmark any model that has been uploaded to the Hugging Face Hub, with reproducible results. Using this package, we benchmark more than 100 Scandinavian or multilingual models and present the results of these in an interactive online leaderboard, as well as provide an analysis of the results. The analysis shows that there is substantial cross-lingual transfer among the Mainland Scandinavian languages (Danish, Swedish and Norwegian), with limited cross-lingual transfer between the group of Mainland Scandinavian languages and the group of Insular Scandinavian languages (Icelandic and Faroese). The benchmarking results also show that the investment in language technology in Norway, Sweden and Denmark has led to language models that outperform massively multilingual models such as XLM-RoBERTa and mDeBERTaV3. We release the source code for both the package and leaderboard.

Motivation & Objective

• To address the growing challenge of model selection and progress evaluation in Scandinavian NLP by creating a standardized benchmark for monolingual and multilingual models.
• To develop a reproducible, accessible evaluation framework that supports any model hosted on the Hugging Face Hub.
• To investigate cross-lingual transfer capabilities between Mainland and Insular Scandinavian languages, given their linguistic and cultural similarities and differences.
• To release two new, high-quality datasets—ScaLA for linguistic acceptability and ScandiQA for question answering—specifically for the Mainland Scandinavian languages.
• To establish an interactive online leaderboard to track and compare model performance across Danish, Norwegian, and Swedish, facilitating transparent progress tracking in the field.

Proposed method

• The benchmark uses four core NLP tasks: named entity recognition, sentiment classification, linguistic acceptability (ScaLA), and question answering (ScandiQA), all uniformly processed for cross-lingual consistency.
• A new Python package, `scandeval`, and CLI interface were developed to enable reproducible, automated benchmarking of any Hugging Face-hosted model on the defined tasks.
• All datasets were standardized and released on the Hugging Face Hub to ensure consistent evaluation across models and languages.
• Cross-lingual transfer was evaluated using F-statistics on performance differences across language groups, comparing models fine-tuned on one language and tested on others.
• The benchmark supports both Mainland (Danish, Norwegian, Swedish) and Insular (Icelandic, Faroese) Scandinavian languages, though the leaderboard focuses on Mainland languages due to observed performance trends.
• Model performance was evaluated using standard metrics: F1 for NER, accuracy for sentiment, and exact match and F1 for question answering, with results aggregated and visualized on an interactive online leaderboard.

Figure 1: Plot showing the performance of different models on the AngryTweets dataset with varying number of training samples.

Experimental results

Research questions

• RQ1 To what extent does cross-lingual transfer occur between the Mainland Scandinavian languages (Danish, Norwegian, Swedish)?
• RQ2 Is there significant cross-lingual transfer between the Mainland Scandinavian languages and the Insular Scandinavian languages (Icelandic, Faroese)?
• RQ3 Do national, monolingual Scandinavian language models outperform general-purpose multilingual models like XLM-RoBERTa and mDeBERTaV3 on Scandinavian NLP tasks?
• RQ4 How does the performance of models trained on one Scandinavian language generalize to other Scandinavian languages, particularly in low-resource settings?
• RQ5 Can a unified benchmarking framework with standardized datasets and reproducible evaluation improve transparency and progress tracking in Scandinavian NLP research?

Key findings

• The Norwegian model, NB-BERT-large, achieved the top overall score and ranked in the top two across all three Mainland Scandinavian languages, indicating strong cross-lingual transfer.
• The best-performing model on Swedish was KB-BERT-large, while DFM-encoder-large-v1 led in Danish, both outperforming general multilingual models like XLM-RoBERTa and mDeBERTaV3.
• F-statistics showed the highest cross-lingual transfer within the Mainland Scandinavian group (F = 33.34 for Danish, Norwegian, and Swedish), confirming substantial linguistic similarity and transferability.
• Cross-lingual transfer between Mainland and Insular Scandinavian languages was minimal, with F-statistics below 10 for all combinations involving both groups, supporting the distinction between the two language clusters.
• The benchmark revealed that national models developed in Norway, Sweden, and Denmark consistently outperform massively multilingual models on their respective languages, suggesting that targeted investment in national language technology yields superior results.
• The online leaderboard, hosted at https://scandeval.github.io, provides real-time, reproducible results for over 100 models, enabling transparent and standardized evaluation across the community.

Figure 2: Boxplot showing the training time of the models on the AngryTweets dataset with varying number of training samples.