[Paper Review] Character-Aware Decoder for Neural Machine Translation

Adithya Renduchintala, Pamela Shapiro+2 more|arXiv (Cornell University)|Sep 6, 2018

Natural Language Processing Techniques21 references4 citations

TL;DR

This paper proposes a character-aware decoder for neural machine translation that enhances word-level and subword-level modeling by integrating convolutional neural networks (CNNs) into the softmax and embedding layers to capture morphological patterns. In low-resource settings, it achieves consistent BLEU gains of up to +3.37 across 14 typologically diverse languages by combining character-aware and standard embeddings on the target side and using an approximate softmax for large vocabularies.

ABSTRACT

Standard neural machine translation (NMT) systems operate primarily on words, ignoring lower-level patterns of morphology. We present a character-aware decoder for NMT that can simultaneously work with both word-level and subword-level sequences which is designed to capture such patterns. We achieve character-awareness by augmenting both the softmax and embedding layers of an attention-based encoder-decoder network with convolutional neural networks that operate on spelling of a word (or subword). While character-aware embeddings have been successfully used in the source-side, we find that mixing character-aware embeddings with standard embeddings is crucial in the target-side. Furthermore, we show that a simple approximate softmax layer can be used for large target-side vocabularies which would otherwise require prohibitively large memory. We experiment on the TED multi-target dataset, translating English into 14 typologically diverse languages. We find that in this low-resource setting, the character-aware decoder provides consistent improvements over word-level and subword-level counterparts with BLEU score gains of up to +3.37.

Motivation & Objective

• To address the limitation of standard NMT models that ignore morphological patterns by operating only on words.
• To improve low-resource neural machine translation by incorporating character-level information into the target-side decoder.
• To reduce memory costs in large-vocabulary translation tasks through an approximate softmax layer.
• To demonstrate the effectiveness of combining character-aware and standard embeddings on the target side for better generalization.

Proposed method

• Augment the attention-based encoder-decoder architecture with CNNs that process the character sequence of words or subwords in both embedding and softmax layers.
• Integrate character-aware embeddings with standard word embeddings on the target side to improve modeling of morphological variations.
• Use a character-level CNN to generate context-aware representations from the spelling of each target word or subword.
• Apply an approximate softmax layer to handle large target-side vocabularies efficiently, reducing memory requirements.
• Train the model end-to-end on the TED multi-target dataset, translating English into 14 diverse languages.
• Optimize the model using standard NMT objectives with cross-entropy loss and beam search decoding.

Experimental results

Research questions

• RQ1 Can character-aware modeling in the decoder improve translation performance in low-resource settings?
• RQ2 How does combining character-aware and standard embeddings on the target side affect translation quality?
• RQ3 Does using an approximate softmax layer maintain performance while reducing memory usage for large vocabularies?
• RQ4 How effective is the character-aware decoder across typologically diverse languages?
• RQ5 What is the impact of character-level modeling on morphologically rich languages in neural machine translation?

Key findings

• The character-aware decoder achieves consistent BLEU score improvements over both word-level and subword-level baselines across 14 diverse languages.
• The model gains up to +3.37 BLEU points in low-resource settings, demonstrating strong generalization across language families.
• Combining character-aware and standard embeddings on the target side yields the most significant performance gains.
• The approximate softmax layer effectively reduces memory usage without degrading translation quality on large vocabularies.
• The method is particularly effective for morphologically rich languages, where character-level modeling captures inflectional patterns better than word-level models.