[Paper Review] Multilingual is not enough: BERT for Finnish

Antti Virtanen, Jenna Kanerva+6 more|arXiv (Cornell University)|Dec 15, 2019

Topic ModelingComputer Science45 references121 citations

TL;DR

The paper evaluates multilingual BERT (M-BERT) versus a Finnish-specific BERT (FinBERT) trained from scratch, showing FinBERT consistently outperforms M-BERT across Finnish POS tagging, NER, and dependency parsing, and achieves new state-of-the-art results. It also examines downstream text classification and probing tasks to characterize language-specific benefits and data-domain effects.

ABSTRACT

Deep learning-based language models pretrained on large unannotated text corpora have been demonstrated to allow efficient transfer learning for natural language processing, with recent approaches such as the transformer-based BERT model advancing the state of the art across a variety of tasks. While most work on these models has focused on high-resource languages, in particular English, a number of recent efforts have introduced multilingual models that can be fine-tuned to address tasks in a large number of different languages. However, we still lack a thorough understanding of the capabilities of these models, in particular for lower-resourced languages. In this paper, we focus on Finnish and thoroughly evaluate the multilingual BERT model on a range of tasks, comparing it with a new Finnish BERT model trained from scratch. The new language-specific model is shown to systematically and clearly outperform the multilingual. While the multilingual model largely fails to reach the performance of previously proposed methods, the custom Finnish BERT model establishes new state-of-the-art results on all corpora for all reference tasks: part-of-speech tagging, named entity recognition, and dependency parsing. We release the model and all related resources created for this study with open licenses at https://turkunlp.org/finbert .

Motivation & Objective

• Assess the effectiveness of multilingual BERT (M-BERT) on Finnish NLP tasks.
• Develop and pretrain a Finnish-specific BERT (FinBERT) from scratch.
• Systematically compare FinBERT and M-BERT on POS tagging, NER, and dependency parsing.
• Evaluate FinBERT and M-BERT on text classification and probing tasks.
• Release FinBERT and associated resources for open research use.

Proposed method

• Pretrain FinBERT (cased and uncased, 110M params) from a large, filtered Finnish corpus combining News, Discussion, crawl data, and Wikipedia.
• Construct dedicated Finnish vocabularies (50k pieces) using BPE on cleaned data with both cased and uncased variants.
• Train on 1M steps per model variant with sequence lengths 128 and 512, using LAMB optimizer with warmup; approximate 12 days per model on 8x Nvidia V100 GPUs.
• Fine-tune both FinBERT and M-BERT on Finnish datasets for POS tagging, NER, dependency parsing, and text classification, using grid search over learning rates and epochs.
• Evaluate using UD Finnish corpora (TDT, FTB, PUD) for POS/Parsing and FiNER for NER; compare against state-of-the-art baselines.

Experimental results

Research questions

• RQ1 Does a language-specific Finnish BERT outperform the multilingual BERT on core Finnish NLP tasks?
• RQ2 To what extent does FinBERT improve POS tagging, NER, and dependency parsing over M-BERT and prior methods?
• RQ3 How does FinBERT perform on Finnish text classification and probing tasks across different data regimes?
• RQ4 What is the impact of tokenization vocabulary and casing on Finnish BERT performance?
• RQ5 How does domain match between pretraining data and evaluation data affect model performance?

Key findings

• FinBERT consistently outperforms M-BERT and prior methods on all three core tasks (POS tagging, NER, dependency parsing) across multiple Finnish corpora.
• On POS tagging, FinBERT cased improves over the best CoNLL’18 results by up to 1.7 percentage points; uncased FinBERT is competitive with cased in several cases.
• On NER, FinBERT (both variants) exceeds M-BERT and FiNER-tagger in in-domain, with FinBERT cased achieving F1 of 81.47 in-domain; FinBERT outperforms M-BERT in out-of-domain as well.
• On dependency parsing, Udify with FinBERT achieves state-of-the-art LAS on all three Finnish treebanks, with gains of 2.3–3.6 percentage points over prior bests; cased FinBERT generally outperforms uncased.
• For text classification, FinBERT outperforms M-BERT, especially with small training sets; domain mismatch (news vs. discussion) influences gains, with FinBERT benefiting from pretraining data that matches informal Finnish.