[Paper Review] GottBERT: a pure German Language Model

Raphael Scheible, Fabian Thomczyk+3 more|arXiv (Cornell University)|Dec 3, 2020

Topic Modeling22 references41 citations

TL;DR

GottBERT is the first German single-language RoBERTa model pre-trained on the German OSCAR corpus and outperforms several German and multilingual models on NER tasks and some text classification tasks.

ABSTRACT

Lately, pre-trained language models advanced the field of natural language processing (NLP). The introduction of Bidirectional Encoders for Transformers (BERT) and its optimized version RoBERTa have had significant impact and increased the relevance of pre-trained models. First, research in this field mainly started on English data followed by models trained with multilingual text corpora. However, current research shows that multilingual models are inferior to monolingual models. Currently, no German single language RoBERTa model is yet published, which we introduce in this work (GottBERT). The German portion of the OSCAR data set was used as text corpus. In an evaluation we compare its performance on the two Named Entity Recognition (NER) tasks Conll 2003 and GermEval 2014 as well as on the text classification tasks GermEval 2018 (fine and coarse) and GNAD with existing German single language BERT models and two multilingual ones. GottBERT was pre-trained related to the original RoBERTa model using fairseq. All downstream tasks were trained using hyperparameter presets taken from the benchmark of German BERT. The experiments were setup utilizing FARM. Performance was measured by the $F_{1}$ score. GottBERT was successfully pre-trained on a 256 core TPU pod using the RoBERTa BASE architecture. Even without extensive hyper-parameter optimization, in all NER and one text classification task, GottBERT already outperformed all other tested German and multilingual models. In order to support the German NLP field, we publish GottBERT under the AGPLv3 license.

Motivation & Objective

• Motivate the creation of a monolingual German RoBERTa model to overcome multilingual model limitations in German NLP.
• Pre-train GottBERT on a large German OSCAR dataset to enable strong downstream performance.
• Evaluate GottBERT against German and multilingual baselines on NER and text classification tasks.

Proposed method

• Pre-train GottBERT using RoBERTa BASE architecture on 145GB of German OSCAR data with a 52k subword vocabulary.
• Use fairseq for pre-training with 256-core TPU pod, 100k update steps, batch size 8k, 10k warmup and polynomial LR decay.
• Convert GottBERT to Hugging Face format for downstream tasks using FARM presets aligned with German BERT benchmarks.
• Evaluate on CoNLL 2003 and GermEval 2014 NER tasks, GermEval 2018 (coarse and fine) and GNAD text classification tasks.
• Measure performance with F1 scores, reporting best-of-10 runs per task based on validation performance.

Experimental results

Research questions

• RQ1 Can a German monolingual RoBERTa model trained on OSCAR data outperform existing German BERT models and multilingual models on standard NLP benchmarks?
• RQ2 What impact does language-specific tokenization and data size have on performance for German NER and text classification tasks?
• RQ3 Are downstream task hyperparameters from German BERT benchmarks sufficient for RoBERTa-based German models, or is further tuning beneficial?

Key findings

• GottBERT outperforms all other tested German and multilingual models on both CoNLL 2003 and GermEval 2014 NER tasks.
• GottBERT achieves best results in GermEval 2018 fine classification, but not consistently across all text classification tasks where XLM RoBERTa or other baselines can outperform.
• Overall, GottBERT shows strong performance even without extensive hyperparameter optimization, highlighting benefits of large German OSCAR data and monolingual modeling.
• GottBERT is the first published German single-language RoBERTa model trained on TPU pods, using a 52k BPE vocabulary derived from German OSCAR text.
• The German OSCAR portion used (145GB) contributes to superior NER results relative to some baselines.
• RoBERTa-based models in the comparison did not always reach full potential, possibly due to sub-optimal downstream hyperparameters.