[Paper Review] IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

Bryan Wilie, Karissa Vincentio+9 more|arXiv (Cornell University)|Sep 11, 2020

Topic Modeling32 references192 citations

TL;DR

IndoNLU introduces the first large Indonesian NLU benchmark with 12 tasks, a new self-supervised Indo4B corpus, and IndoBERT/IndoBERT-lite baselines, plus a benchmark framework and leaderboard for reproducible evaluation.

ABSTRACT

Although Indonesian is known to be the fourth most frequently used language over the internet, the research progress on this language in the natural language processing (NLP) is slow-moving due to a lack of available resources. In response, we introduce the first-ever vast resource for the training, evaluating, and benchmarking on Indonesian natural language understanding (IndoNLU) tasks. IndoNLU includes twelve tasks, ranging from single sentence classification to pair-sentences sequence labeling with different levels of complexity. The datasets for the tasks lie in different domains and styles to ensure task diversity. We also provide a set of Indonesian pre-trained models (IndoBERT) trained from a large and clean Indonesian dataset Indo4B collected from publicly available sources such as social media texts, blogs, news, and websites. We release baseline models for all twelve tasks, as well as the framework for benchmark evaluation, and thus it enables everyone to benchmark their system performances.

Motivation & Objective

• Address resource scarcity in Indonesian NLP by providing a diverse, multi-task benchmark for Indonesian natural language understanding (NLU).
• Create a large, cleaned Indonesian pretraining dataset (Indo4B) to train language models tailored to Indonesian.
• Offer baseline model solutions (IndoBERT and IndoBERT-lite) and compare them against multilingual baselines to establish strong benchmarks.
• Provide a reproducible benchmark framework and an accessible leaderboard to promote transparency and community participation.

Proposed method

• Assemble 12 NLU tasks spanning single-sentence and sentence-pair inputs, including classification and sequence labeling, across formal and colloquial Indonesian.
• Pretrain monolingual Indonesian BERT- and ALBERT-like models (IndoBERT and IndoBERT-lite) on Indo4B using both 128- and 512-token sequence lengths in two-phase training on TPUv3-8.
• Evaluate a broad set of baselines (Scratch, fastText variants, Multilingual BERT, XLM-R, XLM-MLM) withfine-tuning across all tasks using macro F1 for classification and sequence labeling.
• Standardize task splits to enable reproducible comparisons and release a public leaderboard for IndoNLU results.
• Provide preprocessing and vocabulary choices (SentencePiece with BPE, Indonesian-specific handling, privacy masking for Twitter data) to maximize data quality for Indonesian.

Experimental results

Research questions

• RQ1 Can a diverse, multi-task Indonesian NLU benchmark be established to standardize evaluation across models and domains?
• RQ2 Do monolingual Indonesian models (IndoBERT/IndoBERT-lite) outperform multilingual models on Indonesian NLU tasks, and in what settings?
• RQ3 How does Indo4B compare to larger but noisier corpora (e.g., CC-ID) for pretraining word embeddings, and what is the impact on downstream tasks?
• RQ4 What is the trade-off between model size, speed (FLOPs), and accuracy for Indonesian NLU models?
• RQ5 Does increasing input sequence length improve performance on sequence labeling tasks?

Key findings

• IndoBERT-LARGE and XLM-R-LARGE achieve top performance on classification tasks; XLM-R-LARGE and IndoBERT-LARGE lead on sequence labeling tasks.
• Pretrained contextual models outperform scratch and word-embedding baselines across most tasks, illustrating the value of pretraining for Indonesian.
• IndoBERT models often surpass multilingual baselines on classification tasks, with IndoBERT-lite offering competitive results at smaller size.
• Indo4B, despite being smaller than CC-ID-based datasets, yields better downstream performance than fastText trained on CC-ID.
• Multilingual models excel on certain sequence labeling tasks (e.g., NERGrit, FacQA) due to handling borrowed foreign terms, while monolingual models excel in sentence-level semantics.
• Longer input sequences (512 tokens) improve performance on sequence labeling tasks, suggesting benefit from extended context.