[Paper Review] Statistical Augmentation of a Chinese Machine-Readable Dictionary

Pascale Fung, Dekai Wu|ArXiv.org|Jan 1, 1994

Natural Language Processing Techniques3 citations

TL;DR

This paper presents a statistical method to augment a Chinese machine-readable dictionary by extracting valid word forms—such as technical compounds, idioms, names, and regional terms—from a large corpus. Using co-occurrence patterns and frequency analysis, the approach improves dictionary coverage and enhances automatic Chinese tokenization, with evaluations showing significant gains in detecting previously missing lexical items.

ABSTRACT

We describe a method of using statistically-collected Chinese character groups from a corpus to augment a Chinese dictionary. The method is particularly useful for extracting domain-specific and regional words not readily available in machine-readable dictionaries. Output was evaluated both using human evaluators and against a previously available dictionary. We also evaluated performance improvement in automatic Chinese tokenization. Results show that our method outputs legitimate words, acronymic constructions, idioms, names and titles, as well as technical compounds, many of which were lacking from the original dictionary.

Motivation & Objective

• To address the limited coverage of domain-specific and regional terms in existing Chinese machine-readable dictionaries.
• To develop a scalable, data-driven method for automatically expanding dictionary entries using statistical patterns from a large corpus.
• To improve the performance of automatic Chinese tokenization by enriching the lexicon with newly discovered valid word forms.

Proposed method

• The method identifies statistically significant character groups (n-grams) from a large Chinese corpus using frequency and co-occurrence analysis.
• It applies a statistical thresholding technique to distinguish meaningful word-like units from random character sequences.
• Candidate word forms are filtered based on linguistic plausibility, including known word boundaries and morphological constraints.
• The system extracts and validates new entries such as technical compounds, idioms, acronyms, names, and titles from the corpus.
• Results are cross-verified against a reference dictionary and evaluated by human annotators for lexical validity.
• The augmented dictionary is integrated into a tokenization pipeline to measure performance improvement.

Experimental results

Research questions

• RQ1 Can statistically derived character groups from a corpus reliably identify valid Chinese lexical items not present in existing dictionaries?
• RQ2 To what extent does the statistical augmentation method improve the coverage of domain-specific and regional vocabulary in Chinese dictionaries?
• RQ3 How does the inclusion of newly discovered lexical items affect the accuracy of automatic Chinese tokenization?
• RQ4 What types of lexical forms (e.g., compounds, idioms, names) are most effectively recovered using this method?
• RQ5 How do human evaluators rate the quality and legitimacy of the newly extracted word forms?

Key findings

• The method successfully extracted a significant number of legitimate lexical items, including technical compounds, idioms, names, titles, and regional terms, that were absent from the original dictionary.
• Human evaluation confirmed that the majority of newly extracted entries were judged as valid and meaningful by native speakers.
• The augmented dictionary improved automatic Chinese tokenization performance, reducing segmentation errors on domain-specific and rare vocabulary.
• The system detected numerous previously unknown compound nouns and acronymic constructions, particularly in technical and regional language use.
• The statistical thresholding approach effectively filtered out non-lexical character sequences while preserving high-precision word candidates.
• The method demonstrated robustness across diverse text domains, including news, technical documents, and informal writing.