[Paper Review] Neural Machine Translation via Binary Code Prediction

Yusuke Oda, Philip Arthur+3 more|arXiv (Cornell University)|Apr 23, 2017

Natural Language Processing Techniques22 references3 citations

TL;DR

This paper proposes a neural machine translation model that replaces standard softmax output layers with binary code prediction, reducing memory usage and decoding time by up to 10x while maintaining BLEU scores close to those of full softmax. The method uses hybrid prediction (softmax for high-frequency words and binary codes for low-frequency words) and error-correcting codes to improve robustness, achieving faster CPU inference and significantly reduced parameter counts.

ABSTRACT

In this paper, we propose a new method for calculating the output layer in neural machine translation systems. The method is based on predicting a binary code for each word and can reduce computation time/memory requirements of the output layer to be logarithmic in vocabulary size in the best case. In addition, we also introduce two advanced approaches to improve the robustness of the proposed model: using error-correcting codes and combining softmax and binary codes. Experiments on two English-Japanese bidirectional translation tasks show proposed models achieve BLEU scores that approach the softmax, while reducing memory usage to the order of less than 1/10 and improving decoding speed on CPUs by x5 to x10.

Motivation & Objective

• To address the high memory and computational cost of standard softmax output layers in neural machine translation, especially with large vocabularies.
• To develop a method that reduces output layer computation and memory to O(log V) instead of O(V), enabling efficient deployment on resource-constrained systems.
• To improve robustness of binary code prediction, which otherwise suffers from bit-level errors in word representation.
• To maintain competitive translation quality (BLEU) while drastically reducing model size and inference time.
• To enable efficient mini-batching and GPU optimization for training, while ensuring fast CPU inference for production use.

Proposed method

• The model replaces the standard softmax output layer with a binary code representation, where each word is encoded as a binary vector of length log₂V, reducing computation and parameter count to O(log V).
• A hybrid prediction model is introduced, where high-frequency words are predicted via standard softmax, and low-frequency words are predicted via learned binary codes, improving overall accuracy.
• Error-correcting codes (specifically convolutional codes) are applied to the binary representations, enabling Viterbi decoding to correct bit errors and enhance robustness.
• The model uses a two-stage prediction: first predicting individual bits of the binary code, then decoding the final word using either Viterbi decoding (for error-correcting codes) or direct mapping (for simple binary codes).
• The training objective minimizes cross-entropy between the true one-hot word vector and the predicted probability distribution derived from the binary code predictions.
• The method is designed to be compatible with mini-batching and efficient GPU training, while enabling fast CPU inference due to reduced output layer complexity.

Experimental results

Research questions

• RQ1 Can binary code prediction in the output layer reduce memory and computation costs in NMT while maintaining competitive BLEU scores?
• RQ2 How effective is the combination of hybrid prediction (softmax + binary codes) in improving accuracy compared to pure binary code prediction?
• RQ3 Can error-correcting codes significantly improve the robustness of binary code prediction in NMT models?
• RQ4 To what extent does the proposed method reduce inference time and memory usage on CPUs compared to standard softmax?
• RQ5 Is the proposed method compatible with efficient mini-batching and GPU training for large-scale NMT systems?

Key findings

• The proposed Binary-EC model reduces output layer memory usage to less than 1/10 of standard softmax, achieving a 10x reduction in parameter count.
• On CPU inference, the proposed models achieve 5x to 10x faster decoding speed compared to standard softmax, with some configurations reaching up to 20x speedup.
• The Hybrid-N-EC model achieves BLEU scores comparable to or higher than standard softmax on the BTEC English-Japanese translation task, despite using far fewer parameters.
• The simple binary code model alone performs poorly (significantly lower BLEU), demonstrating that robustness is critical and that error correction and hybrid prediction are essential for performance.
• The hybrid model with error correction (Binary-EC) achieves higher BLEU than the hybrid model without error correction, even with 1/10 the number of parameters, showing that redundancy improves accuracy more than incremental prediction.
• The trade-off curve between BLEU and model size shows that for easier datasets (e.g., BTEC), a smaller softmax layer (N ≤ 1024) is sufficient, while harder datasets (e.g., ASPEC) require larger layers to saturate performance.