[Paper Review] IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP

Fajri Koto, Afshin Rahimi+2 more|arXiv (Cornell University)|Nov 2, 2020

Topic ModelingComputer Science50 references32 citations

TL;DR

The paper releases IndoLEM, a comprehensive Indonesian NLP benchmark with seven tasks, and IndoBERT, a monolingual Indonesian BERT model that achieves state-of-the-art results on IndoLEM tasks compared to baselines and multilingual models.

ABSTRACT

Although the Indonesian language is spoken by almost 200 million people and the 10th most spoken language in the world, it is under-represented in NLP research. Previous work on Indonesian has been hampered by a lack of annotated datasets, a sparsity of language resources, and a lack of resource standardization. In this work, we release the IndoLEM dataset comprising seven tasks for the Indonesian language, spanning morpho-syntax, semantics, and discourse. We additionally release IndoBERT, a new pre-trained language model for Indonesian, and evaluate it over IndoLEM, in addition to benchmarking it against existing resources. Our experiments show that IndoBERT achieves state-of-the-art performance over most of the tasks in IndoLEM.

Motivation & Objective

• Provide a standardized, comprehensive Indonesian NLP benchmark dataset with diverse tasks (morpho-syntax, semantics, discourse).
• Release a monolingual Indonesian BERT model (IndoBERT) trained on large Indonesian corpora.
• Benchmark IndoBERT against multilingual models and existing Indonesian resources to establish state-of-the-art baselines.

Proposed method

• Introduce IndoLEM: seven NLP tasks across three categories (morpho-syntax/sequence labeling, semantics, discourse) with standardized splits and metrics.
• Develop IndoBERT: a BERT-style Indonesian language model trained on 220M Indonesian words from Wikipedia, news, and web corpora, using a 12-layer, 768-hidden architecture and Indonesian WordPiece vocabulary.
• Evaluate IndoBERT on IndoLEM against baselines including BiLSTM with fastText embeddings and other BERT models (mBERT, MalayBERT).
• Integrate IndoBERT into a BiAffine dependency parser and evaluate on UD-Indo-GSD and UD-Indo-PUD datasets.
• Create two discourse tasks (Next Tweet Prediction and Tweet Ordering) based on Indonesian Twitter data to assess discourse coherence.
• Provide detailed evaluation methodologies for each task (cross-validation, F1, accuracy, ROUGE, Spearman ρ).

Experimental results

Research questions

• RQ1 Can a standardized Indonesian benchmark (IndoLEM) catalyze progress in Indonesian NLP by providing comparable splits and metrics across multiple tasks?
• RQ2 Does a monolingual Indonesian BERT model (IndoBERT) outperform existing multilingual and Indonesian-specific baselines on diverse NLP tasks?
• RQ3 How do IndoBERT representations affect performance across morpho-syntax, semantic, and discourse tasks in IndoLEM?
• RQ4 Do discourse-related tasks (Next Tweet Prediction, Tweet Ordering) reveal advantages of IndoBERT over other models?
• RQ5 What is the impact of integrating contextual embeddings into standard parsers for Indonesian dependency parsing?

Key findings

• IndoBERT achieves state-of-the-art performance on most IndoLEM tasks compared to baselines and competing BERT models.
• IndoBERT improves sentiment analysis and summarization more substantially than other models, with +13.2 F1 over naive Bayes and +7.5 over mBERT for sentiment analysis, and ~1–2 ROUGE points for summarization.
• IndoBERT yields the best results for POS tagging and NER among the BERT-based models, and strong improvements in UD-Indo-GSD dependency parsing with BiAffine parser.
• On UD-Indo-PUD, mBERT outperforms IndoBERT in dependency parsing, highlighting dataset-specific effects and translation quality issues in UD-Indo-PUD.
• Discourse tasks show IndoBERT achieving the highest accuracy in Next Tweet Prediction and the highest Spearman ρ in Tweet Ordering among tested models, with human performance (oracle) and observed gaps noted.
• IndoLEM provides a large, standardized Indonesian benchmark with novel discourse tasks, driving reproducible evaluation and progress in Indonesian NLP.