[Paper Review] BERTje: A Dutch BERT Model

Wietse de Vries, Andreas van Cranenburgh+4 more|arXiv (Cornell University)|Dec 19, 2019

Topic Modeling21 references214 citations

TL;DR

BERTje is a monolingual Dutch BERT model trained on diverse Dutch text sources, consistently outperforming multilingual BERT on Dutch NLP tasks such as NER, POS tagging, SRL, STR, and sentiment analysis.

ABSTRACT

The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at https://github.com/wietsedv/bertje.

Motivation & Objective

• Motivate the need for a high-quality monolingual Dutch BERT model beyond multilingual BERT.
• Build and pre-train a Dutch BERT-base model on diverse, large-scale Dutch data.
• Evaluate BERTje on a suite of Dutch NLP tasks to compare against multilingual BERT.
• Analyze how pre-training iterations affect performance across low- and high-level linguistic tasks.

Proposed method

• Use BERT-base architecture with 12 transformer blocks (BERT-base).
• Assemble a diverse Dutch pre-training corpus totaling ~12GB/2.4B tokens from books, TwNC, SoNaR-500, web news, and Wikipedia (with overlap removal).
• Create a 30k WordPiece vocabulary via SentencePiece adapted to WordPiece format.
• Pre-train with SOP (replacing NSP) and masked language modeling masking 15% of tokens, where 80%→[MASK], 10%→random, 10%→unchanged, with consecutive pieces masked to cover full words.
• Fine-tune on multiple Dutch NLP tasks (NER, POS, SRL, STR, sentiment) and compare to multilingual BERT base.
• Evaluate at 850k and 1M training iterations to assess learning dynamics.

Experimental results

Research questions

• RQ1 Does a monolingual Dutch BERT model trained on diverse Dutch data outperform the multilingual BERT on Dutch-specific tasks?
• RQ2 How does pre-training iteration count (850k vs 1M) affect performance across low- and high-level linguistic tasks in Dutch?
• RQ3 Which Dutch NLP tasks benefit most from a monolingual Dutch BERT model?

Key findings

• BERTje outperforms multilingual BERT on NER in both CoNLL-2002 and SoNaR-1 datasets (e.g., NER F1 improves from 80.7 to 88.3 on test with full BERTje).
• On POS tagging, BERTje achieves higher accuracy than multilingual BERT across Lassy Small and SoNaR-1 (test) datasets, with final results around mid-to-high 96s.
• For semantic roles and spatio-temporal relations, BERTje higher than multilingual BERT (SRL and STR), with gains most pronounced at the 850k checkpoint and some improvement persisting at 1M.
• In sentiment analysis on the Dutch Book Reviews dataset, BERTje approaches or matches state-of-the-art performance without extensive hyperparameter tuning (fully trained BERTje achieving 93.0% test).
• The 850k checkpoint often matches or closely approaches the fully trained BERTje, indicating earlier encoding of useful information for several tasks, while some high-level tasks benefit from longer pre-training.