[Paper Review] Transfer Deep Learning for Low-Resource Chinese Word Segmentation with a Novel Neural Network

Jingjing Xu, Xu Sun|arXiv (Cornell University)|Feb 15, 2017

Natural Language Processing Techniques26 references3 citations

TL;DR

This paper proposes a transfer learning framework using a teacher-student neural network to improve low-resource Chinese word segmentation. By pre-training a teacher model on high-resource data and initializing the student model with its weights, then applying a weighted data similarity method to adapt to low-resource datasets, the approach achieves state-of-the-art F-scores of 96.1% on PKU and 96.2% on CTB, with a 2.3% and 1.5% F-score gain over prior methods.

ABSTRACT

Recent studies have shown effectiveness in using neural networks for Chinese word segmentation. However, these models rely on large-scale data and are less effective for low-resource datasets because of insufficient training data. We propose a transfer learning method to improve low-resource word segmentation by leveraging high-resource corpora. First, we train a teacher model on high-resource corpora and then use the learned knowledge to initialize a student model. Second, a weighted data similarity method is proposed to train the student model on low-resource data. Experiment results show that our work significantly improves the performance on low-resource datasets: 2.3% and 1.5% F-score on PKU and CTB datasets. Furthermore, this paper achieves state-of-the-art results: 96.1%, and 96.2% F-score on PKU and CTB datasets.

Motivation & Objective

• Address the challenge of poor performance in Chinese word segmentation on low-resource datasets due to insufficient labeled training data.
• Leverage high-resource corpora (e.g., MSR) to improve model generalization on low-resource datasets like PKU and CTB.
• Develop a transfer learning framework that transfers knowledge from a pre-trained teacher model to a student model to reduce reliance on random initialization.
• Introduce a weighted data similarity method to handle distribution shifts between high- and low-resource corpora during student model training.
• Accelerate training through mini-batch asynchronous parallel (MAP) learning without sacrificing model performance.

Proposed method

• Train a teacher model on a high-resource corpus (e.g., MSR) using a novel unified global-local neural network architecture.
• Initialize the student model with the trained weights from the teacher model to provide a better starting point than random initialization.
• Compute weighted data similarity between each high-resource sample and the low-resource dataset using precision and recall to determine relevance.
• Update the learning rate for each sample adaptively based on the error rate $ e^t = 1 - \frac{2p^t r^t}{p^t + r^t} $ and the update rate $ a^t = \frac{1}{2} \log \frac{1 - e^t}{e^t} $.
• Apply mini-batch asynchronous parallel (MAP) learning to accelerate training, reducing training time by nearly 5x compared to serial training.
• Use a unified global-local neural network that combines bidirectional LSTM and gated recursive networks for enhanced feature extraction.

Experimental results

Research questions

• RQ1 Can transfer learning from high-resource corpora significantly improve Chinese word segmentation performance on low-resource datasets?
• RQ2 How can distribution shifts between high- and low-resource corpora be mitigated during knowledge transfer?
• RQ3 Does adaptive learning rate scheduling based on data similarity improve model convergence and performance on low-resource data?
• RQ4 Can mini-batch asynchronous parallel learning be effectively applied to neural word segmentation to accelerate training without performance degradation?
• RQ5 To what extent does the proposed method outperform existing state-of-the-art models on low-resource benchmarks like PKU and CTB?

Key findings

• The proposed method achieves state-of-the-art F-scores of 96.1% on the PKU dataset and 96.2% on the CTB dataset, outperforming prior neural models.
• On low-resource datasets, the method improves F-score by 2.3 percentage points on PKU and 1.5 percentage points on CTB compared to previous SOTA methods.
• Transfer learning using the teacher-student framework improves F-score by 1.0% on PKU and 0.5% on CTB when transferring knowledge from the MSR corpus.
• The weighted data similarity method effectively reduces distribution shift, enabling better knowledge transfer from high-resource to low-resource data.
• Mini-batch asynchronous parallel learning accelerates training by nearly 5x compared to serial training while maintaining identical F-score performance.
• The model achieves a 34.3% reduction in error rate on PKU and 26.3% on CTB compared to baseline Bi-LSTM models, demonstrating strong generalization on low-resource data.