[Paper Review] KLUE: Korean Language Understanding Evaluation

Sungjoon Park, Jihyung Moon+29 more|arXiv (Cornell University)|May 20, 2021

Topic Modeling145 references78 citations

TL;DR

KLUE introduces eight Korean NLU tasks built from scratch with diverse, accessible corpora and ethical annotation protocols; baseline models show Korean-specific PLMs excel, PII removal has minimal impact, and morpheme-level tokenization helps in morpheme-sensitive tasks.

ABSTRACT

We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU) tasks, including Topic Classification, SemanticTextual Similarity, Natural Language Inference, Named Entity Recognition, Relation Extraction, Dependency Parsing, Machine Reading Comprehension, and Dialogue State Tracking. We build all of the tasks from scratch from diverse source corpora while respecting copyrights, to ensure accessibility for anyone without any restrictions. With ethical considerations in mind, we carefully design annotation protocols. Along with the benchmark tasks and data, we provide suitable evaluation metrics and fine-tuning recipes for pretrained language models for each task. We furthermore release the pretrained language models (PLM), KLUE-BERT and KLUE-RoBERTa, to help reproducing baseline models on KLUE and thereby facilitate future research. We make a few interesting observations from the preliminary experiments using the proposed KLUE benchmark suite, already demonstrating the usefulness of this new benchmark suite. First, we find KLUE-RoBERTa-large outperforms other baselines, including multilingual PLMs and existing open-source Korean PLMs. Second, we see minimal degradation in performance even when we replace personally identifiable information from the pretraining corpus, suggesting that privacy and NLU capability are not at odds with each other. Lastly, we find that using BPE tokenization in combination with morpheme-level pre-tokenization is effective in tasks involving morpheme-level tagging, detection and generation. In addition to accelerating Korean NLP research, our comprehensive documentation on creating KLUE will facilitate creating similar resources for other languages in the future. KLUE is available at https://klue-benchmark.com.

Motivation & Objective

• Cover diverse Korean NLU aspects across formal and colloquial text.
• Ensure accessibility by using freely redistributable corpora and permissive licenses.
• Design precise, unambiguous annotation protocols and mitigate biases and privacy concerns.
• Provide evaluation metrics and fine-tuning recipes for pretrained language models on each task.
• Release KLUE baseline models to facilitate replication and future research.

Proposed method

• Select eight NLU tasks (TC, STS, NLI, NER, RE, DP, MRC, DST) to cover varied linguistic phenomena.
• Construct task-specific datasets from ten source corpora under open licenses and filter for quality and ethics.
• Preprocess data with Korean Sentence Splitter and automated filters for noise, toxicity, and PII.
• Annotate with Korean-centric guidelines to reflect Korean morphology and syntax.
• Define task-appropriate evaluation metrics for each task (e.g., macro F1, Pearson, accuracy, UAS/LAS, EM, ROUGE-W, joint goal accuracy).
• Propose and release strong baselines based on Korean PLMs (KLUE-BERT, KLUE-RoBERTa, KoELECTRA, etc.) and report fine-tuning configurations.

Experimental results

Research questions

• RQ1 How can a diverse, open Korean NLU benchmark be built from scratch to avoid translation artifacts?
• RQ2 What are the baseline performances of Korean-specific PLMs on the KLUE tasks?
• RQ3 How does removing PII from pretraining data affect downstream NLU performance across tasks?
• RQ4 Which tokenization and pre-tokenization strategies best handle Korean morphology in NLU tasks?
• RQ5 What ethical safeguards and annotation procedures yield high-quality, unbiased Korean NLU annotations?

Key findings

• KLUE-RoBERTa_LARGE outperforms other baselines across tasks, including multilingual PLMs and open-source Korean PLMs.
• KOELECTRA_BASE excels on STS and NLI tasks among the smaller models, while KLUE-BERT dominates TC and WoS tasks in certain configurations.
• KLUE-RoBERTa_LARGE generally surpasses other models as size increases, with some exceptions in NER.
• Removing PII from pretraining data shows minimal degradation in downstream task performance.
• Morpheme-based subword tokenization combined with morpheme-level pre-tokenization is effective for morpheme-level tagging, detection, and generation.
• KLUE provides comprehensive task documentation to enable future language resources in other languages.