[Paper Review] Neural Networks for Text Correction and Completion in Keyboard Decoding

Shaona Ghosh, Per Ola Kristensson|arXiv (Cornell University)|Sep 19, 2017

Natural Language Processing Techniques28 references56 citations

TL;DR

The paper presents a character-based CNN-GRU encoder with a word-level GRU attention decoder (CCEAD) for simultaneous text correction and completion, trained end-to-end on a combined Twitter typo/OpenSubtitles-derived dataset, achieving competitive CER and word accuracy with a much smaller footprint than traditional language models.

ABSTRACT

Despite the ubiquity of mobile and wearable text messaging applications, the problem of keyboard text decoding is not tackled sufficiently in the light of the enormous success of the deep learning Recurrent Neural Network (RNN) and Convolutional Neural Networks (CNN) for natural language understanding. In particular, considering that the keyboard decoders should operate on devices with memory and processor resource constraints, makes it challenging to deploy industrial scale deep neural network (DNN) models. This paper proposes a sequence-to-sequence neural attention network system for automatic text correction and completion. Given an erroneous sequence, our model encodes character level hidden representations and then decodes the revised sequence thus enabling auto-correction and completion. We achieve this by a combination of character level CNN and gated recurrent unit (GRU) encoder along with and a word level gated recurrent unit (GRU) attention decoder. Unlike traditional language models that learn from billions of words, our corpus size is only 12 million words; an order of magnitude smaller. The memory footprint of our learnt model for inference and prediction is also an order of magnitude smaller than the conventional language model based text decoders. We report baseline performance for neural keyboard decoders in such limited domain. Our models achieve a word level accuracy of $90\%$ and a character error rate CER of $2.4\%$ over the Twitter typo dataset. We present a novel dataset of noisy to corrected mappings by inducing the noise distribution from the Twitter data over the OpenSubtitles 2009 dataset; on which our model predicts with a word level accuracy of $98\%$ and sequence accuracy of $68.9\%$. In our user study, our model achieved an average CER of $2.6\%$ with the state-of-the-art non-neural touch-screen keyboard decoder at CER of $1.6\%$.

Motivation & Objective

• Motivate the need for on-device, resource-efficient text correction and completion for mobile keyboards.
• Propose an end-to-end sequence-to-sequence model that jointly corrects errors and completes text.
• Leverage a character-level encoder and a word-level decoder with attention to handle noise and provide language modeling effects.
• Demonstrate effectiveness on real and synthetic noisy data while maintaining a small memory footprint.

Proposed method

• Propose a two-part architecture: a character-based CNN-GRU encoder that captures error representations, and a word-based GRU attention decoder that acts as an implicit language model.
• Use a joint end-to-end training regime on OpenTypo, a combination of Twitter typos and OpenSubtitles-derived text, to model noisy-to-correct mappings.
• Employ a sequence-to-sequence framework where input is a noisy character sequence and output is a corrected word sequence, with attention-based context from encoder to decoder.
• Encode characters with CNN filters of widths 2–4 and 5 filters, GRU size 256, and feed into a single-layer word-level GRU decoder of size 256.
• Initialize the decoder from the encoder’s final state and apply attention over encoder states to form context vectors for generation.
• Train with cross-entropy loss over the word vocabulary (approx. 3,000 words in the decoder) and use Adam optimization.

Experimental results

Research questions

• RQ1 Can a character-level CNN-GRU encoder paired with a word-level attention decoder jointly correct typos and complete text on resource-constrained devices?
• RQ2 Does end-to-end training on a mixed noisy dataset improve robustness to real user input compared to traditional spell-check + language model baselines?
• RQ3 What are the trade-offs in accuracy versus model footprint for on-device keyboard decoding tasks?
• RQ4 How does the proposed model perform on real user data and on open-domain noisy text datasets compared to state-of-the-art non-neural decoders?

Key findings

• The model achieves a word-level accuracy of 90% and a character error rate (CER) of 2.4 on the Twitter typo dataset.
• On a novel OpenSubtitles-derived OpenTypo dataset, the model reaches 98% word accuracy and sequence accuracy of 0.689.
• User study with 8 users shows CER around 2.6% on real inputs, competitive with state-of-the-artCER of 1.6% for non-neural touch-screen decoders and 2.1% on physical keyboards.
• The model is an order of magnitude smaller in memory footprint, training time, and corpus size compared to conventional language model-based decoders.
• Training and datasets (including the OpenTypo data) are released as open-source tools for inference and benchmarking.