[Paper Review] Choosing the Right Word: Using Bidirectional LSTM Tagger for Writing Support Systems

Victor Makarenkov, Lior Rokach+1 more|arXiv (Cornell University)|Jan 8, 2019

Topic Modeling38 references4 citations

TL;DR

This paper proposes a bidirectional LSTM tagger for automated word choice assistance in writing support systems, specifically targeting English as a second language (ESL) learners. By leveraging large-scale, unlabeled text corpora without manual annotation, the model learns contextually appropriate word substitutions, outperforming state-of-the-art methods in both domain-specific (scientific writing) and general-purpose writing tasks, with key contributions including a novel unsupervised approach and publicly shared code, models, and an ESL test set.

ABSTRACT

Scientific writing is difficult. It is even harder for those for whom English is a second language (ESL learners). Scholars around the world spend a significant amount of time and resources proofreading their work before submitting it for review or publication. In this paper we present a novel machine learning based application for proper word choice task. Proper word choice is a generalization the lexical substitution (LS) and grammatical error correction (GEC) tasks. We demonstrate and evaluate the usefulness of applying bidirectional Long Short Term Memory (LSTM) tagger, for this task. While state-of-the-art grammatical error correction uses error-specific classifiers and machine translation methods, we demonstrate an unsupervised method that is based solely on a high quality text corpus and does not require manually annotated data. We use a bidirectional Recurrent Neural Network (RNN) with LSTM for learning the proper word choice based on a word's sentential context. We demonstrate and evaluate our application on both a domain-specific (scientific), writing task and a general-purpose writing task. We show that our domain-specific and general-purpose models outperform state-of-the-art general context learning. As an additional contribution of this research, we also share our code, pre-trained models, and a new ESL learner test set with the research community.

Motivation & Objective

• To address the challenge of proper word choice in writing, especially for ESL learners who struggle with grammatical errors and lexical substitutions.
• To develop a unified, unsupervised model that generalizes across both grammatical error correction (GEC) and lexical substitution (LS) tasks.
• To eliminate reliance on manually annotated training data, enabling scalable deployment without costly human labeling.
• To create a practical, real-world writing support tool that suggests contextually accurate word replacements based on linguistic patterns.
• To contribute a new, realistic ESL learner test set, pre-trained models, and open-source code to the NLP research community.

Proposed method

• A bidirectional Long Short-Term Memory (LSTM) network is trained on large monolingual text corpora to learn contextual representations of words.
• The model predicts the most probable correct word in a given sentence context, treating word choice as a tagging problem.
• The approach uses only unlabeled, high-quality text data—no human-annotated error or substitution labels are required.
• The model is fine-tuned and evaluated on two settings: a domain-specific scientific writing corpus and a general-purpose corpus (COCA).
• Word replacement suggestions are generated based on the model’s predicted probability distribution over the vocabulary for each masked word.
• Evaluation uses Mean Reciprocal Rank (MRR) on human-edited test sets to measure ranking quality of suggested corrections.

Experimental results

Research questions

• RQ1 Can a single, unsupervised bidirectional LSTM model effectively handle both grammatical error correction and lexical substitution tasks in word choice?
• RQ2 How does the performance of the model compare to state-of-the-art supervised GEC and LS systems on both general and domain-specific writing?
• RQ3 To what extent does the model generalize across different writing styles, such as scientific versus general-purpose English?
• RQ4 How effective is the model in suggesting correct words when the original sentence contains subtle collocation or word form errors?
• RQ5 Can the model be applied without any human-annotated training data while still achieving competitive performance?

Key findings

• The domain-specific model achieved an MRR of 0.61 on the combined correct list test set, significantly outperforming the lower bound of a single-correct-option model.
• The general-purpose model achieved an MRR of 0.49 on the intersection test set, demonstrating strong performance on diverse word choice tasks.
• On a manually collected ESL test set, the domain-specific model achieved an MRR of 0.27, showing practical utility in real-world writing scenarios.
• The model outperformed state-of-the-art context-learning models in both scientific and general writing evaluations.
• The system demonstrated robustness in handling various error types, including collocation, word form, and article usage, without requiring error-specific classifiers.
• The authors successfully shared a new, realistic ESL learner test set, pre-trained models, and code, enabling reproducibility and further research.