[Paper Review] Neural Chinese Word Segmentation with Dictionary Knowledge

Junxin Liu, Fangzhao Wu+3 more|arXiv (Cornell University)|Jul 11, 2018

Natural Language Processing Techniques14 references4 citations

TL;DR

This paper proposes two end-to-end methods to enhance neural Chinese word segmentation (CWS) by integrating dictionary knowledge: pseudo-labeled data generation and multi-task learning with a word classification task. Experiments show significant performance gains, especially under low-resource conditions, demonstrating that dictionary integration improves OOV word handling and reduces reliance on large labeled datasets.

ABSTRACT

Chinese word segmentation (CWS) is an important task for Chinese NLP. Recently, many neural network based methods have been proposed for CWS. However, these methods require a large number of labeled sentences for model training, and usually cannot utilize the useful information in Chinese dictionary. In this paper, we propose two methods to exploit the dictionary information for CWS. The first one is based on pseudo labeled data generation, and the second one is based on multi-task learning. The experimental results on two benchmark datasets validate that our approach can effectively improve the performance of Chinese word segmentation, especially when training data is insufficient.

Motivation & Objective

• To address the limitation of neural CWS models in handling out-of-vocabulary (OOV) words due to insufficient training data.
• To leverage existing Chinese dictionaries to enhance model generalization without manual feature engineering.
• To develop end-to-end trainable methods that integrate dictionary information directly into neural CWS frameworks.
• To reduce dependency on large-scale labeled datasets by incorporating external lexical knowledge.

Proposed method

• Generate pseudo-labeled training sentences by randomly sampling words from a Chinese dictionary and combining them into valid sequences.
• Train a neural CWS model on both real labeled sentences and the generated pseudo-labeled sentences, with a learnable weight (λ₁) to balance their contributions.
• Introduce a multi-task learning setup where a separate word classification head is trained alongside CWS to predict whether a character sequence forms a valid Chinese word.
• Share the lower-level neural network parameters between the CWS and word classification tasks to enable joint optimization.
• Use both internal dictionaries (built from training data) and external dictionaries (e.g., Sogou) to improve coverage and model robustness.
• Optimize the model with a weighted loss combining CWS and word classification objectives, where λ₂ controls the relative importance of the auxiliary task.

Experimental results

Research questions

• RQ1 Can dictionary knowledge improve neural CWS performance when labeled training data is limited?
• RQ2 Does incorporating dictionary information through pseudo-labeled data generation lead to better generalization on OOV words?
• RQ3 Can multi-task learning with a word classification head enhance CWS performance compared to standard neural models?
• RQ4 How do internal and external dictionaries compare in improving segmentation accuracy?
• RQ5 What is the optimal balance between real labeled data, pseudo-labeled data, and auxiliary word classification signals?

Key findings

• The proposed methods significantly improve CWS performance on benchmark datasets, especially when training data is scarce.
• Using an external dictionary (e.g., Sogou) leads to better performance than relying solely on internal dictionaries.
• Combining both internal and external dictionaries yields the best results, indicating complementary information.
• Performance improves with increasing dictionary size, confirming that broader lexical coverage enhances model learning.
• Optimal hyperparameters λ₁ and λ₂ exist—too high values degrade performance by overemphasizing pseudo or auxiliary signals.
• Case studies confirm that the model correctly segments OOV words like “被害人” and rare phrases like “一口气” that standard models fail to segment.