[Paper Review] Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks

Dan Saattrup Nielsen, Kenneth Enevoldsen+1 more|arXiv (Cornell University)|Jun 19, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper extends the ScandEval benchmark to evaluate decoder language models on multilingual NLU tasks across eight Germanic languages, demonstrating that decoder models significantly outperform encoder models in NLU, particularly on question-answering tasks, with performance varying by language and task type. The study reveals distinct learning trajectories between encoder and decoder models via UMAP analysis.

ABSTRACT

This paper explores the performance of encoder and decoder language models on multilingual Natural Language Understanding (NLU) tasks, with a broad focus on Germanic languages. Building upon the ScandEval benchmark, initially restricted to evaluating encoder models, we extend the evaluation framework to include decoder models. We introduce a method for evaluating decoder models on NLU tasks and apply it to the languages Danish, Swedish, Norwegian, Icelandic, Faroese, German, Dutch, and English. Through a series of experiments and analyses, we also address research questions regarding the comparative performance of encoder and decoder models, the impact of NLU task types, and the variation across language resources. Our findings reveal that encoder models can achieve significantly better NLU performance than decoder models despite having orders of magnitude fewer parameters. Additionally, we investigate the correlation between decoders and task performance via a UMAP analysis, shedding light on the unique capabilities of decoder and encoder models. This study contributes to a deeper understanding of language model paradigms in NLU tasks and provides valuable insights for model selection and evaluation in multilingual settings.

Motivation & Objective

• To address the lack of systematic comparison between encoder and decoder language models in multilingual NLU settings, especially beyond English.
• To extend the ScandEval benchmark to include few-shot evaluation of decoder models and broaden language coverage to include German, Dutch, and English.
• To investigate whether decoder models achieve superior NLU performance compared to state-of-the-art fine-tuned encoder models across diverse tasks and languages.
• To analyze how model type (encoder vs. decoder) correlates with performance variation across NLU task types and language resource levels.

Proposed method

• Extended the ScandEval benchmarking framework to support few-shot evaluation of decoder models, enabling consistent comparison with encoder models.
• Adapted decoder models for NLU tasks by prompting them with instruction-tuned and zero-shot formats, ensuring compatibility with standard NLU evaluation protocols.
• Applied UMAP dimensionality reduction to visualize and compare the performance distribution of encoder and decoder models across leaderboards.
• Collected and published model scores on public leaderboards for 8 Germanic languages, including metadata on model size, inference speed, and weight availability.
• Evaluated a wide range of models, including BERT-based encoders, RoBERTa, XLM-RoBERTa, and decoder models such as Llama-3, GPT-4, and GPT-3.5-turbo.
• Used mean rank score as the primary metric, stratified by task type and model generative capability, to assess performance differences.

Figure 1: UMAP plots of the models on the ScandEval leaderboards.

Experimental results

Research questions

• RQ1 Can state-of-the-art decoder models achieve significantly better NLU performance than state-of-the-art fine-tuned encoder models?
• RQ2 Does the performance advantage of decoder models depend on the type of NLU task?
• RQ3 Does the performance difference between encoder and decoder models vary across the language resource spectrum, from low- to high-resource languages?

Key findings

• Decoder models significantly outperform encoder models on multilingual NLU tasks, with the best-performing decoder models achieving mean rank scores up to 1.40 on Danish and 1.46 on English, compared to 1.33 and 1.34 for encoder models.
• The performance advantage of decoder models is most pronounced on question-answering tasks, where generative models show strong correlation with high performance, even without instruction tuning.
• Decoder models exhibit poor performance on named entity recognition and linguistic acceptability tasks, indicating task-specific biases in their capabilities.
• UMAP analysis reveals distinct performance trajectories for encoder and decoder models, suggesting they learn different representations during training.
• The performance gap between encoder and decoder models varies by language, with stronger advantages observed in low-resource languages like Faroese and Icelandic.
• The study confirms that decoder models are heavily biased toward question-answering tasks, with non-instruction-tuned models still showing strong performance, likely due to architectural or pre-training factors.

Figure 2: The correlation between a model being generative and its performance on the four NLU tasks.