[Paper Review] An unsupervised and customizable misspelling generator for mining noisy health-related text sources

Abeed Sarker, Graciela Gonzalez‐Hernandez|arXiv (Cornell University)|Jun 3, 2018

Topic Modeling29 references61 citations

TL;DR

This paper presents an unsupervised, data-centric misspelling generator that leverages dense word embeddings and semantic-lexical filtering to automatically produce common, semantically relevant misspellings of complex health-related terms. The system outperforms state-of-the-art methods with an F1-score of 0.69 and increases Twitter retrieval rates by over 67% when applied to cancer-related terms.

ABSTRACT

In this paper, we present a customizable datacentric system that automatically generates common misspellings for complex health-related terms. The spelling variant generator relies on a dense vector model learned from large unlabeled text, which is used to find semantically close terms to the original/seed keyword, followed by the filtering of terms that are lexically dissimilar beyond a given threshold. The process is executed recursively, converging when no new terms similar (lexically and semantically) to the seed keyword are found. Weighting of intra-word character sequence similarities allows further problem-specific customization of the system. On a dataset prepared for this study, our system outperforms the current state-of-the-art for medication name variant generation with best F1-score of 0.69 and F1/4-score of 0.78. Extrinsic evaluation of the system on a set of cancer-related terms showed an increase of over 67% in retrieval rate from Twitter posts when the generated variants are included. Our proposed spelling variant generator has several advantages over the current state-of-the-art and other types of variant generators-(i) it is capable of filtering out lexically similar but semantically dissimilar terms, (ii) the number of variants generated is low as many low-frequency and ambiguous misspellings are filtered out, and (iii) the system is fully automatic, customizable and easily executable. While the base system is fully unsupervised, we show how supervision maybe employed to adjust weights for task-specific customization. The performance and significant relative simplicity of our proposed approach makes it a much needed misspelling generation resource for health-related text mining from noisy sources. The source code for the system has been made publicly available for research purposes.

Motivation & Objective

• To address the challenge of missing relevant health-related data in noisy text sources like social media due to frequent misspellings of complex medical terms.
• To develop a fully automatic, unsupervised system that generates only semantically and lexically plausible misspellings, avoiding irrelevant variants.
• To create a customizable, plug-and-play tool for researchers in health NLP without requiring NLP or machine learning expertise.
• To improve data collection efficiency by reducing false positives and minimizing API keyword limits through precision-controlled variant generation.

Proposed method

• The system uses a pre-trained dense vector model (word embeddings) learned from large unlabeled text to find semantically similar terms to a seed keyword.
• It applies recursive filtering to retain only terms that are both semantically close (via embedding similarity) and lexically similar (via edit distance) to the seed.
• A threshold-based filtering mechanism ensures that only variants within a specified lexical and semantic distance are retained.
• The system incorporates customizable weighting of intra-word character sequence similarities to fine-tune precision and recall for specific use cases.
• The method is fully unsupervised by default but supports lightweight supervision to adjust weights for task-specific needs.
• The algorithm is implemented in Python and designed for easy deployment and customization without external dependencies or manual curation.

Experimental results

Research questions

• RQ1 Can a data-centric, unsupervised approach generate more accurate and relevant spelling variants for complex medical terms than existing phonetic or rule-based methods?
• RQ2 How effective is the system in retrieving real-world health-related content from noisy sources like Twitter when using generated misspellings?
• RQ3 To what extent can semantic and lexical filtering reduce false positives in spelling variant generation while preserving high recall?
• RQ4 Can the system be customized to prioritize precision or recall based on application-specific requirements without retraining?
• RQ5 How does the performance of the system vary across different medical domains, such as medication names and disease terms?

Key findings

• The system achieved an F1-score of 0.69 on a benchmark dataset of 20 medication names, outperforming the current state-of-the-art.
• On an extrinsic evaluation using cancer-related terms, the inclusion of generated variants increased retrieval rates from Twitter by over 67%.
• The system significantly reduces false positives by filtering out lexically similar but semantically unrelated terms, improving precision.
• The weighted variant generation mode successfully reduced false positives for closely related medications like duloxetine and paroxetine.
• The system generated a small, manageable number of high-quality variants, making it suitable for use with keyword-limited APIs such as Twitter’s.
• Error analysis revealed that low-frequency misspellings not well-represented in the embedding space were the main cause of low recall, suggesting that larger or domain-specific training data could further improve performance.