[Paper Review] IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization

Fajri Koto, Jey Han Lau+1 more|arXiv (Cornell University)|Sep 10, 2021

Topic Modeling27 references4 citations

TL;DR

This paper introduces IndoBERTweet, the first large-scale pretrained language model for Indonesian Twitter, which enhances a monolingual Indonesian BERT model by adding domain-specific vocabulary through averaging subword embeddings. The method achieves 5× faster pretraining and outperforms both pretraining from scratch and word2vec-based initialization across seven Twitter-based NLP tasks, demonstrating superior efficiency and effectiveness in domain adaptation.

ABSTRACT

We present IndoBERTweet, the first large-scale pretrained model for Indonesian Twitter that is trained by extending a monolingually-trained Indonesian BERT model with additive domain-specific vocabulary. We focus in particular on efficient model adaptation under vocabulary mismatch, and benchmark different ways of initializing the BERT embedding layer for new word types. We find that initializing with the average BERT subword embedding makes pretraining five times faster, and is more effective than proposed methods for vocabulary adaptation in terms of extrinsic evaluation over seven Twitter-based datasets.

Motivation & Objective

• To develop a high-performing, large-scale pretrained language model tailored for Indonesian social media text.
• To address vocabulary mismatch in domain-adaptive pretraining by evaluating multiple initialization strategies for domain-specific words.
• To reduce the computational cost of domain adaptation while maintaining or improving model performance.
• To establish a benchmark for Indonesian Twitter NLP by releasing a domain-specific, state-of-the-art model.

Proposed method

• Extends IndoBERT, a monolingual Indonesian BERT model, by adding domain-specific vocabulary tokens from a 26M-tweet Indonesian Twitter corpus.
• Applies WordPiece tokenization to the Twitter dataset and constructs a domain-specific vocabulary of 31,984 types.
• Implements domain-adaptive pretraining by fine-tuning the base IndoBERT model on the Twitter corpus for 200K steps.
• Tests four vocabulary initialization strategies: random, word2vec projection, and two variants of subword embedding averaging (mean and learned).
• Uses masked language modeling objective with a maximum sequence length of 128 tokens, following the original IndoBERT configuration.
• Evaluates performance across seven downstream Twitter-based NLP tasks, including sentiment analysis and named entity recognition.

Experimental results

Research questions

• RQ1 Does initializing domain-specific vocabulary with the average of subword embeddings from the base model improve performance over random or word2vec-based initialization?
• RQ2 How does the computational efficiency of domain-adaptive pretraining compare to pretraining from scratch when using different vocabulary initialization methods?
• RQ3 Can a domain-adaptive approach with subword-averaged initialization outperform a model pretraining from scratch on Indonesian Twitter data?
• RQ4 How does vocabulary mismatch affect model performance, and to what extent can subword averaging mitigate this issue?

Key findings

• The subword embedding averaging method reduced pretraining time by 80% compared to training from scratch, achieving a fivefold speedup.
• On average, the subword-averaged initialization method achieved the highest F1 score across seven downstream tasks, outperforming both random and word2vec-based initialization.
• The model initialized with averaged subword embeddings achieved a performance gap of only -0.2% absolute compared to the original IndoBERT, indicating minimal degradation without pretraining.
• After 200K steps of domain-adaptive pretraining, the subword-averaged method outperformed even the model pretraining from scratch for 1M steps, demonstrating superior sample efficiency.
• The method showed strong generalization, with the average number of subwords per new token being 2.6 for IndoBERTweet, compared to 3.4 for BERTweet, indicating better tokenization efficiency.
• In English transfer experiments, the subword-averaged method narrowed the performance gap between RoBERTa and BERTweet from +3.4% to +2.2% absolute, despite using only 1/10th the pretraining data.