[Paper Review] MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining

Wei Zhu|arXiv (Cornell University)|Nov 17, 2020

Topic Modeling28 references4 citations

TL;DR

This paper proposes MVP-BERT, a novel approach to redesign Chinese BERT's vocabulary using Chinese word segmentation and subword tokenization (seg_tok), and introduces three multi-vocabulary pretraining (MVP) strategies—hierarchical, pairwise, and objective-based—to enhance model expressiveness. Experiments show that seg_tok improves performance on sentence-level tasks and efficiency, while MVP strategies, especially MVP_obj and MVP_pair, significantly boost performance on sequence labeling tasks like NER, with MVP_obj achieving near-competitive results using less than half the training time.

ABSTRACT

Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary for these Chinese PLMs remain to be the one provided by Google Chinese Bert \cite{devlin2018bert}, which is based on Chinese characters. Second, the masked language model pre-training is based on a single vocabulary, which limits its downstream task performances. In this work, we first propose a novel method, \emph{seg\_tok}, to form the vocabulary of Chinese BERT, with the help of Chinese word segmentation (CWS) and subword tokenization. Then we propose three versions of multi-vocabulary pretraining (MVP) to improve the models expressiveness. Experiments show that: (a) compared with char based vocabulary, \emph{seg\_tok} does not only improves the performances of Chinese PLMs on sentence level tasks, it can also improve efficiency; (b) MVP improves PLMs' downstream performance, especially it can improve \emph{seg\_tok}'s performances on sequence labeling tasks.

Motivation & Objective

• To address the limitations of character-based vocabularies in Chinese BERT, which can hinder semantic learning and efficiency.
• To explore alternative vocabulary designs that balance word-level semantics and subword flexibility for improved downstream performance.
• To develop multi-vocabulary pretraining (MVP) strategies that enhance model expressiveness by integrating multiple vocabularies during pretraining.
• To evaluate the effectiveness of seg_tok and MVP strategies across diverse Chinese NLP benchmarks, especially on sequence labeling tasks.
• To provide a resource-efficient alternative to single-vocabulary models by analyzing trade-offs between performance and training cost.

Proposed method

• Proposes seg_tok, a hybrid vocabulary construction method combining Chinese word segmentation (CWS) with subword tokenization to create a more semantically meaningful vocabulary than character-level tokenization.
• Introduces MVP_hier, a hierarchical multi-vocabulary strategy that aggregates character embeddings into word-level representations before feeding them into the transformer encoder.
• Designs MVP_pair, a dual-encoder approach with shared parameters, where two different vocabularies (e.g., char and seg_tok) are used simultaneously in masked language modeling, with loss weighted by a coefficient λ.
• Develops MVP_obj, a single-encoder strategy that applies a coarse-grained vocabulary (e.g., seg_tok) as an auxiliary objective to predict full words from subword representations, enhancing semantic learning.
• Employs whole-word masking during pretraining to ensure coherent prediction of multi-character words, improving alignment with linguistic units.
• Uses a relative importance coefficient λ to balance the loss contributions from different vocabularies in MVP_pair, enabling ablation studies on optimal trade-offs.

Experimental results

Research questions

• RQ1 Can a vocabulary based on segmented words (seg_tok) outperform the standard character-based vocabulary in Chinese BERT for sentence-level NLP tasks?
• RQ2 How does multi-vocabulary pretraining (MVP) improve the expressiveness and downstream performance of Chinese PLMs compared to single-vocabulary pretraining?
• RQ3 What is the impact of different MVP strategies—hierarchical, pairwise, and objective-based—on model accuracy and training efficiency?
• RQ4 How does the relative importance coefficient λ affect the performance of MVP_pair, and what optimal settings exist for different downstream tasks?
• RQ5 Can MVP_obj achieve performance close to MVP_pair with significantly reduced training cost, especially in resource-constrained settings?

Key findings

• The seg_tok vocabulary improves performance on sentence-level tasks (e.g., LCQMC, MSRA) by up to 2.46% over character-based BERT, with faster inference due to shorter sequences.
• MVP_obj with (char, seg_tok) achieves 86.10% accuracy on LCQMC and 73.95% on NER, outperforming single-vocab models and approaching the performance of full ensemble models.
• MVP_pair with both encoders (e, ftc) achieves 87.96% on LCQMC and 80.05% on MSRA, demonstrating that ensemble training enhances performance, especially on NER.
• The coarse-to-fine MVP_pair (char → seg_tok) outperforms fine-to-coarse (seg_tok → char) on NER, indicating that coarse-grained supervision aids in learning robust representations.
• MVP_obj with λ=2.0 achieves near-competitive results to MVP_pair with less than half the training time, making it a more efficient alternative in low-resource scenarios.
• Increasing λ beyond 1.0 degrades performance on NER for MVP_pair(s), but improves it on sentence-level tasks, indicating task-dependent optimal λ values.