[Paper Review] Conceptualized Representation Learning for Chinese Biomedical Text Mining

Ningyu Zhang, Qianghuai Jia+4 more|arXiv (Cornell University)|Aug 25, 2020

Topic Modeling11 references55 citations

TL;DR

MC-BERT introduces whole entity and whole span masking to inject biomedical knowledge into Chinese BERT pretraining, releasing ChineseBLUE benchmark and achieving state-of-the-art results on Chinese biomedical NLP tasks.

ABSTRACT

Biomedical text mining is becoming increasingly important as the number of biomedical documents and web data rapidly grows. Recently, word representation models such as BERT has gained popularity among researchers. However, it is difficult to estimate their performance on datasets containing biomedical texts as the word distributions of general and biomedical corpora are quite different. Moreover, the medical domain has long-tail concepts and terminologies that are difficult to be learned via language models. For the Chinese biomedical text, it is more difficult due to its complex structure and the variety of phrase combinations. In this paper, we investigate how the recently introduced pre-trained language model BERT can be adapted for Chinese biomedical corpora and propose a novel conceptualized representation learning approach. We also release a new Chinese Biomedical Language Understanding Evaluation benchmark ( extbf{ChineseBLUE}). We examine the effectiveness of Chinese pre-trained models: BERT, BERT-wwm, RoBERTa, and our approach. Experimental results on the benchmark show that our approach could bring significant gain. We release the pre-trained model on GitHub: https://github.com/alibaba-research/ChineseBLUE.

Motivation & Objective

• Motivate adapting BERT to Chinese biomedical text due to domain-specific word distributions and long-tail terminologies.
• Propose MC-BERT with knowledge-infused pretraining to inject biomedical concepts into representations.
• Release ChineseBLUE, a large-scale Chinese Biomedical Language Understanding Evaluation benchmark for multiple tasks.

Proposed method

• Pretrain MC-BERT starting from BERT-base on a biomedical Chinese corpus.
• Use whole entity masking to mask complete biomedical entities sourced from a knowledge graph and NER tools.
• Use whole span masking to cover fine-grained biomedical phrases extracted via Autophrase and Biomedical Phrase resources, filtered by rules and fastText.
• Train with a learning rate around 1e-5 without learning rate warmup to avoid catastrophic forgetting.
• Split input into sentence-based positive samples and cross-document negative samples for next-sentence prediction.
• Incorporate sentences, phrases, and entities to guide word embedding learning rather than adding separate knowledge embeddings.

Experimental results

Research questions

• RQ1 Can injecting domain-specific biomedical knowledge into pretraining improve Chinese biomedical NLP tasks?
• RQ2 How do entity-level and span-level masking strategies affect representation learning for Chinese biomedical text?
• RQ3 Does a knowledge-infused pretraining approach yield improvements over standard BERT variants on the Chinese Biomedical Language Understanding Evaluation (ChineseBLUE) tasks?

Key findings

• MC-BERT outperforms BERT-base on several Chinese biomedical tasks in ChineseBLUE.
• Ablation shows performance declines when removing whole entity masking or whole span masking, highlighting the benefit of these strategies.
• Compared to RoBERTa and other Chinese pretrained models, MC-BERT with biomedical pretraining offers superior performance on the evaluated tasks.