[Paper Review] GlyphCRM: Bidirectional Encoder Representation for Chinese Character with its Glyph

Yunxin Li, Yu Zhao+6 more|arXiv (Cornell University)|Jul 1, 2021

Topic Modeling39 references4 citations

TL;DR

GlyphCRM is a novel Chinese pre-trained language model that replaces traditional character embeddings with glyph-based representations, using a residual CNN (HanGlyph) to extract spatial features from character images and integrating them into a bidirectional Transformer via skip connections. It achieves state-of-the-art performance across 9 downstream NLU tasks, outperforming BERT by up to 1% on average, and demonstrates strong generalization on low-resource and specialized domains like medical text.

ABSTRACT

Previous works indicate that the glyph of Chinese characters contains rich semantic information and has the potential to enhance the representation of Chinese characters. The typical method to utilize the glyph features is by incorporating them into the character embedding space. Inspired by previous methods, we innovatively propose a Chinese pre-trained representation model named as GlyphCRM, which abandons the ID-based character embedding method yet solely based on sequential character images. We render each character into a binary grayscale image and design two-channel position feature maps for it. Formally, we first design a two-layer residual convolutional neural network, namely HanGlyph to generate the initial glyph representation of Chinese characters, and subsequently adopt multiple bidirectional encoder Transformer blocks as the superstructure to capture the context-sensitive information. Meanwhile, we feed the glyph features extracted from each layer of the HanGlyph module into the underlying Transformer blocks by skip-connection method to fully exploit the glyph features of Chinese characters. As the HanGlyph module can obtain a sufficient glyph representation of any Chinese character, the long-standing out-of-vocabulary problem could be effectively solved. Extensive experimental results indicate that GlyphCRM substantially outperforms the previous BERT-based state-of-the-art model on 9 fine-tuning tasks, and it has strong transferability and generalization on specialized fields and low-resource tasks. We hope this work could spark further research beyond the realms of well-established representation of Chinese texts.

Motivation & Objective

• To address the limitation of traditional Chinese NLP models that rely on static, ID-based character embeddings, which ignore rich semantic information in character glyphs.
• To explore whether glyph representations alone—without character IDs—can serve as effective, context-aware representations for Chinese text.
• To develop a pre-trained model that leverages both glyph structure and contextual dependencies to improve transferability and generalization, especially in low-resource and specialized domains.
• To solve the out-of-vocabulary (OOV) problem inherent in embedding-based models by relying on visual glyph features instead of vocabulary lookup.

Proposed method

• Each Chinese character is rendered as a binary grayscale image and fed into a two-stage residual convolutional network (HanGlyph) to extract hierarchical glyph representations.
• Two-channel position maps are introduced to encode spatial structure and relative position within the character glyph.
• The glyph features from each HanGlyph block are passed to the underlying Transformer blocks via skip-connections to preserve and utilize multi-scale visual features.
• The model uses a bidirectional Transformer encoder as the superstructure to capture long-range contextual dependencies across characters.
• Pre-training is conducted using the same objectives as BERT: Masked Language Modeling and Next Sentence Prediction, but on image-based inputs only.
• Fine-tuning is performed directly on image representations without requiring character ID lookup, enabling end-to-end adaptation to downstream tasks.

Experimental results

Research questions

• RQ1 Can a pre-trained language model for Chinese that relies solely on glyph images—without character IDs—achieve superior performance compared to ID-based models?
• RQ2 To what extent do glyph-based representations improve generalization and transferability in low-resource and specialized NLP tasks?
• RQ3 Can a glyph-only representation effectively address the out-of-vocabulary problem common in embedding-based models?
• RQ4 How do skip-connections from multi-level glyph features enhance contextual representation in the Transformer encoder?

Key findings

• GlyphCRM outperforms the BERT-based state-of-the-art model on 9 downstream Chinese NLU tasks, with an average improvement of up to 1% in accuracy.
• On the medical semantic matching dataset (CMSSM), GlyphCRM achieves 90.84% test accuracy, surpassing BERT’s 89.79%.
• On a low-resource e-commerce sentiment analysis dataset, GlyphCRM achieves 76.07% test accuracy, outperforming BERT’s 73.61% by 2.44 percentage points.
• The model demonstrates strong transferability in specialized domains, such as medical text, where it generalizes better than BERT despite the presence of rare and domain-specific terms.
• The performance gain is attributed to the intrinsic semantic richness of Chinese character glyphs, which convey meaningful visual patterns linked to word meanings.
• The absence of a character embedding table eliminates the out-of-vocabulary problem, as all characters are represented via their visual forms regardless of frequency or prior occurrence.