[Paper Review] RobBERT-2022: Updating a Dutch Language Model to Account for Evolving Language Use

Pieter Delobelle, Thomas Winters+1 more|Lirias (KU Leuven)|Nov 15, 2022

Topic Modeling4 citations

TL;DR

This paper presents RobBERT-2022, an updated version of the state-of-the-art Dutch RoBERTa-based language model, by extending its tokenizer with new high-frequency tokens from the 2022 Dutch OSCAR corpus and further pre-training on this data. The updated model achieves significant performance gains on recent Dutch NLP tasks, particularly for COVID-19 related text, while maintaining strong performance on older benchmarks, demonstrating the value of continual language model adaptation to evolving language use.

ABSTRACT

Large transformer-based language models, e.g. BERT and GPT-3, outperform previous architectures on most natural language processing tasks. Such language models are first pre-trained on gigantic corpora of text and later used as base-model for finetuning on a particular task. Since the pre-training step is usually not repeated, base models are not up-to-date with the latest information. In this paper, we update RobBERT, a RoBERTa-based state-of-the-art Dutch language model, which was trained in 2019. First, the tokenizer of RobBERT is updated to include new high-frequent tokens present in the latest Dutch OSCAR corpus, e.g. corona-related words. Then we further pre-train the RobBERT model using this dataset. To evaluate if our new model is a plug-in replacement for RobBERT, we introduce two additional criteria based on concept drift of existing tokens and alignment for novel tokens.We found that for certain language tasks this update results in a significant performance increase. These results highlight the benefit of continually updating a language model to account for evolving language use.

Motivation & Objective

• To address the issue of language models becoming outdated due to evolving language use, particularly for Dutch.
• To update RobBERT, a state-of-the-art Dutch BERT model trained in 2019, to reflect recent linguistic changes.
• To evaluate whether the updated model maintains performance on older tasks while improving on newer, more recent ones.
• To introduce new evaluation criteria based on concept drift and alignment for novel tokens.

Proposed method

• Extended the RoBERTa tokenizer to include new high-frequency subwords and tokens from the 2022 Dutch OSCAR corpus, such as corona-related terms.
• Further pre-trained the RobBERT v2 model on the 2022 OSCAR dataset using masked language modeling (MLM) as the sole pre-training objective.
• Used AdamW optimizer with a base 1e-6 initial learning rate and 1,000 warmup steps, applying early stopping based on validation perplexity (PPPL).
• Evaluated the model using a combination of benchmark tasks, recent domain-specific tasks (e.g., COVID-19 FAQ and tweet classification), and t-SNE visualization of sentence embeddings.
• Introduced two new evaluation criteria: concept drift analysis on existing tokens and alignment assessment for novel tokens.

Experimental results

Research questions

• RQ1 Does further pre-training on a more recent Dutch text corpus improve performance on recent NLP tasks without degrading performance on older benchmarks?
• RQ2 To what extent does the updated model exhibit concept drift or catastrophic forgetting on previously learned linguistic patterns?
• RQ3 How well do the embeddings of RobBERT-2022 capture semantic information for newly introduced tokens, such as those related to the COVID-19 pandemic?
• RQ4 Can the updated model serve as a direct plug-in replacement for the original RobBERT in downstream applications?

Key findings

• RobBERT-2022 achieved a 79.3% F1 score on the VaccinChat task for classifying Dutch COVID-19 vaccine FAQs, outperforming the original RobBERT v2 (77.2% F1).
• On a tweet classification task about Belgian COVID-19 measures, RobBERT-2022 achieved 75% accuracy, compared to 73% for both the original RobBERT and multilingual BERT.
• The model showed minimal concept drift, with no significant performance drop on older benchmark tasks, and only a slight increase in pseudo-perplexity (9.40 vs. 7.76) on the original 2019 corpus.
• t-SNE visualization revealed a clear shift in sentence embeddings between RobBERT-2022 and the original model, indicating adaptation to new semantic patterns in the 2022 data.
• The updated model maintains strong generalization ability while showing improved alignment with recent language use, particularly for pandemic-related terminology.