[Paper Review] Glyce: Glyph-vectors for Chinese Character Representations

Yuxian Meng, Wei Wu+8 more|arXiv (Cornell University)|Jan 29, 2019

Natural Language Processing Techniques70 references147 citations

TL;DR

Glyce introduces glyph-based character representations by using historical Chinese scripts and a specialized Tianzige-CNN, combined with auxiliary image-classification objectives and optional integration with BERT, to achieve state-of-the-art results across multiple Chinese NLP tasks.

ABSTRACT

It is intuitive that NLP tasks for logographic languages like Chinese should benefit from the use of the glyph information in those languages. However, due to the lack of rich pictographic evidence in glyphs and the weak generalization ability of standard computer vision models on character data, an effective way to utilize the glyph information remains to be found. In this paper, we address this gap by presenting Glyce, the glyph-vectors for Chinese character representations. We make three major innovations: (1) We use historical Chinese scripts (e.g., bronzeware script, seal script, traditional Chinese, etc) to enrich the pictographic evidence in characters; (2) We design CNN structures (called tianzege-CNN) tailored to Chinese character image processing; and (3) We use image-classification as an auxiliary task in a multi-task learning setup to increase the model's ability to generalize. We show that glyph-based models are able to consistently outperform word/char ID-based models in a wide range of Chinese NLP tasks. We are able to set new state-of-the-art results for a variety of Chinese NLP tasks, including tagging (NER, CWS, POS), sentence pair classification, single sentence classification tasks, dependency parsing, and semantic role labeling. For example, the proposed model achieves an F1 score of 80.6 on the OntoNotes dataset of NER, +1.5 over BERT; it achieves an almost perfect accuracy of 99.8\% on the Fudan corpus for text classification. Code found at https://github.com/ShannonAI/glyce.

Motivation & Objective

• Motivate the use of pictographic, glyph-based information for logographic Chinese NLP tasks.
• Propose Glyce: glyph embeddings derived from multi-script historical images of Chinese characters.
• Develop a CNN architecture (Tianzige-CNN) tailored to small character images and multiple scripts.
• Enhance generalization via an auxiliary image-classification objective in a multi-task setup.
• Demonstrate state-of-the-art performance across tagging, classification, parsing, and SRL tasks, with potential integration with BERT.

Proposed method

• Represent Chinese characters as images and extract glyph embeddings using Tianzige-CNN (田字格) tailored for small character images and group convolutions to reduce overfitting.
• Enrich glyph signals by using an ensemble of historical scripts (bronze ware, seal, clerical, traditional, cursive, etc.) and multiple writing styles.
• Apply an auxiliary objective of image classification to predict the character ID, with a decaying weight throughout training to improve generalization.
• Optionally combine glyph embeddings with BERT by creating a Glyce-BERT pipeline where glyph outputs are fused with BERT representations and processed by transformer layers.
• Use task-specific output layers (often transformers) on the concatenated glyph and BERT embeddings for downstream tasks (sequence labeling, single/sentence-pair classification, parsing, SRL).
• Evaluate on diverse Chinese NLP tasks and compare against non-BERT baselines, BERT, and other state-of-the-art models.

Experimental results

Research questions

• RQ1 Can glyph-based representations derived from historical Chinese scripts improve performance on downstream NLP tasks compared to traditional character/word embeddings?
• RQ2 Does a specialized Tianzige-CNN architecture better capture glyph information for small character images than standard CNNs?
• RQ3 Does multi-task training with an image-classification objective improve generalization of glyph embeddings?
• RQ4 Does glyce-BERT integration yield state-of-the-art results across tagging, classification, parsing, and SRL tasks?
• RQ5 What is the impact of training strategies and CNN architectures on performance when using glyce representations?

Key findings

• Glyce+BERT achieves state-of-the-art results across multiple datasets and tasks, often outperforming BERT alone (e.g., NER OntoNotes F1 of 81.63 vs 79.16 for BERT in Table 2).
• Glyce-based models achieve strong results on CWS, POS, and NER datasets, with Glyce+BERT generally providing the best performance among tested configurations.
• In sentence-pair classification, Glyce+BERT attains new SOTA across BQ, LCQMC, XNLI, and NLPCC-DBQA benchmarks (Table 5).
• On single-sentence classification, Glyce+BERT achieves near-perfect accuracy on the Fudan corpus (99.8%) and outperforms LSTM/Glyce baselines across ChnSentiCorp, the Fudan corpus, and iFeng (Table 6).
• For dependency parsing, Glyce-word improves UAS/LAS by about 0.8–0.9 points over previous SOTA models (Table 7).
• In SRL, Glyce with the k-order pruning backbone yields +0.8 F1 over the current SOTA (83.7 F1, Table 7).
• Ablation studies show that: (i) the BERT-glyce-joint strategy outperforms other strategies; (ii) transformer-based task-specific outputs outperform BiLSTMs/CNNs/BiMPM; (iii) the image-classification auxiliary objective adds roughly +0.8 in performance; (iv) the Tianzige-CNN structure provides a notable +1.0 F1 boost over vanilla CNNs (Tables 8–11).