[Paper Review] Spelling Correction as a Foreign Language

Yingbo Zhou, Utkarsh Porwal+1 more|arXiv (Cornell University)|May 21, 2017

Topic Modeling12 references6 citations

TL;DR

This paper reformulates spelling correction as a neural machine translation task using an encoder-decoder framework with bidirectional LSTMs, treating misspelled queries as 'foreign language' input. The end-to-end model achieves competitive performance (62.5% accuracy) on an internal e-commerce dataset without feature engineering, outperforming prior methods while simplifying the pipeline by replacing separate language and error models.

ABSTRACT

In this paper, we reformulated the spell correction problem as a machine translation task under the encoder-decoder framework. This reformulation enabled us to use a single model for solving the problem that is traditionally formulated as learning a language model and an error model. This model employs multi-layer recurrent neural networks as an encoder and a decoder. We demonstrate the effectiveness of this model using an internal dataset, where the training data is automatically obtained from user logs. The model offers competitive performance as compared to the state of the art methods but does not require any feature engineering nor hand tuning between models.

Motivation & Objective

• To address the limitations of traditional noisy channel models in spelling correction, which require separate language and error models and are sensitive to error model inaccuracies.
• To eliminate the need for manual feature engineering and hyperparameter tuning by unifying spelling correction into a single end-to-end deep learning framework.
• To demonstrate that sequence-to-sequence models with attention can effectively learn conditional probability P(x| x̃) directly from user log data.
• To evaluate the model on a large-scale, real-world e-commerce spelling correction dataset collected from user search session logs.

Proposed method

• Reformulate spelling correction as a sequence-to-sequence learning problem, treating misspelled text as 'source' and correct text as 'target' in a neural machine translation setup.
• Use a bidirectional LSTM encoder to process input characters or BPE-encoded subwords and generate contextual hidden states.
• Employ an attention-augmented LSTM decoder that attends to relevant encoder states to generate the corrected output token by token.
• Train the entire model end-to-end using gradient descent to maximize the likelihood of the correct output given the misspelled input.
• Use beam search during inference to generate the most probable corrected sequence.
• Utilize BPE tokenization to improve performance by capturing subword semantics, reducing sequence length, and improving generalization.

Experimental results

Research questions

• RQ1 Can a single end-to-end neural sequence-to-sequence model outperform traditional two-stage noisy channel models in spelling correction without requiring feature engineering?
• RQ2 How does the choice of input representation (characters vs. BPE subwords) affect model performance on spelling correction tasks?
• RQ3 To what extent can user log data from search sessions be automatically mined to create high-quality, large-scale training data for spelling correction?
• RQ4 Does attention-based decoding improve correction accuracy compared to standard autoregressive decoding in this context?
• RQ5 How robust is the model to diverse and noisy spelling errors found in real-world e-commerce search queries?

Key findings

• The model achieved 62.5% accuracy on a human-evaluated test set, outperforming the prior state-of-the-art method (62.0%) from Hasan et al. (2015).
• The BPE-based encoder and decoder (W-2-W RNN) performed best, suggesting subword units improve modeling of complex spelling variations.
• The character-based decoder (C-2-C) achieved 55.1% accuracy, demonstrating that even low-level representations can be effective with end-to-end training.
• The model's performance was competitive despite not requiring any feature engineering or manual tuning, highlighting the robustness of the end-to-end approach.
• The use of user session logs enabled automatic collection of 70 million training pairs, showing the feasibility of large-scale unsupervised data collection for spelling correction.
• The attention mechanism allowed the model to dynamically focus on relevant parts of the input, improving correction quality in complex cases.