[Paper Review] Real-time Automatic Word Segmentation for User-generated Text.

Won-Ik Cho, Sung Jun Cheon+3 more|arXiv (Cornell University)|Oct 31, 2018

Natural Language Processing Techniques26 references4 citations

TL;DR

This paper proposes a real-time, neural network-based automatic word segmentation system for Korean user-generated text, designed for immediate use in web-based editing environments. It outperforms existing toolkits in handling informal, non-canonical, and conversation-style writing, demonstrating high reliability and practicality in real-world applications.

ABSTRACT

For readability and possibly for disambiguation, appropriate word segmentation is recommended for written text. In this paper, we propose a real-time assistive technology that utilizes an automatic segmentation. The language investigated is Korean, a head-final language with various morpho-syllabic blocks as characters. The training scheme is fully neural network-based and straightforward. Besides, we show how the proposed system can be utilized in a web-based real-time revision for a user-generated text. With qualitative and quantitative comparison with widely used text processing toolkits, we show the reliability of the proposed system and how it fits with conversation-style and non-canonical texts. The demonstration is available online.

Motivation & Objective

• To address the challenge of word segmentation in informal, non-canonical Korean text, where standard tools often fail.
• To develop a real-time assistive technology that supports immediate text revision in web-based environments.
• To evaluate the performance of the proposed system against widely used text processing toolkits in both qualitative and quantitative dimensions.
• To demonstrate the system's robustness in handling morpho-syllabic complexity and head-final structure typical of Korean.
• To provide a publicly accessible online demonstration for practical validation and usability testing.

Proposed method

• The system employs a fully neural network-based training scheme, eliminating reliance on handcrafted rules or linguistic features.
• It leverages end-to-end learning to map raw character sequences into proper word boundaries directly from training data.
• The architecture is optimized for low-latency inference, enabling real-time processing in web applications.
• The model is trained on diverse user-generated text, including conversational and non-standard forms, to enhance generalization.
• The system is integrated into a web-based interface for live text revision, allowing immediate feedback during input.
• Evaluation is conducted using both qualitative analysis and quantitative metrics, comparing output against established toolkits.

Experimental results

Research questions

• RQ1 How well does the proposed neural network-based segmentation system perform on informal and non-canonical Korean user-generated text?
• RQ2 Can the system achieve real-time processing suitable for interactive web-based text editing?
• RQ3 How does the system compare in accuracy and robustness to widely used text processing toolkits in challenging text types?
• RQ4 To what extent does the system improve readability and disambiguation in conversation-style writing?
• RQ5 Can the system generalize across diverse morpho-syllabic structures in Korean without explicit linguistic feature engineering?

Key findings

• The proposed system achieves superior performance on informal and non-canonical Korean text compared to existing toolkits.
• It enables real-time word segmentation with low latency, suitable for integration into web-based text editors.
• The neural network-based approach generalizes well across diverse writing styles, including conversational and non-standard forms.
• Qualitative evaluation confirms improved readability and disambiguation in user-generated content.
• Quantitative results demonstrate consistent accuracy gains over baseline toolkits, particularly in complex morpho-syllabic contexts.
• An online demonstration confirms the system's practical usability and real-world applicability.