[Paper Review] Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation

Xiang Kong, Qizhe Xie+2 more|arXiv (Cornell University)|Sep 25, 2018

Topic Modeling37 references3 citations

TL;DR

This paper proposes using Byte Pair Encoding (BPE) and Hybrid-LightRNN as efficient vocabulary encoding schemes to reduce the memory and computation costs of Mixture of Softmaxes (MoS) in sequence generation. By halving time and memory usage without performance loss, MoS with BPE or Hybrid-LightRNN achieves state-of-the-art results, including 29.6 BLEU on WMT 2014 English-to-German translation and 0.76 CIDEr improvement on image captioning.

ABSTRACT

Mixture of Softmaxes (MoS) has been shown to be effective at addressing the expressiveness limitation of Softmax-based models. Despite the known advantage, MoS is practically sealed by its large consumption of memory and computational time due to the need of computing multiple Softmaxes. In this work, we set out to unleash the power of MoS in practical applications by investigating improved word coding schemes, which could effectively reduce the vocabulary size and hence relieve the memory and computation burden. We show both BPE and our proposed Hybrid-LightRNN lead to improved encoding mechanisms that can halve the time and memory consumption of MoS without performance losses. With MoS, we achieve an improvement of 1.5 BLEU scores on IWSLT 2014 German-to-English corpus and an improvement of 0.76 CIDEr score on image captioning. Moreover, on the larger WMT 2014 machine translation dataset, our MoS-boosted Transformer yields 29.5 BLEU score for English-to-German and 42.1 BLEU score for English-to-French, outperforming the single-Softmax Transformer by 0.8 and 0.4 BLEU scores respectively and achieving the state-of-the-art result on WMT 2014 English-to-German task.

Motivation & Objective

• Address the high memory and computational cost of Mixture of Softmaxes (MoS), which limits its practical use despite improved model expressiveness.
• Reduce vocabulary size through efficient encoding schemes to lower the computational and memory burden of computing multiple Softmax components.
• Improve the efficiency and scalability of MoS for large-scale language generation tasks such as machine translation and image captioning.
• Demonstrate that MoS with BPE or Hybrid-LightRNN can achieve state-of-the-art performance while maintaining or reducing inference cost.
• Investigate whether learned encoding schemes like Hybrid-LightRNN outperform heuristic methods like BPE, especially in handling out-of-vocabulary words and capturing semantic structure.

Proposed method

• Apply Byte Pair Encoding (BPE) to subword-encode words, reducing vocabulary size and enabling more efficient Softmax computation by representing words as sequences of subword units.
• Introduce Hybrid-LightRNN, a learnable encoding mechanism that learns a table-based codebook for words using a language modeling objective, grouping semantically or syntactically similar words into shared rows/columns.
• Use the learned codebook to represent each word as a sequence of code tokens, transforming the Softmax computation into a product of conditional distributions over code sequences.
• Apply MoS with multiple Softmax components over the code sequences, enabling higher-rank probability matrices and improved expressiveness without increasing vocabulary size.
• Optimize the encoding tables end-to-end via backpropagation, allowing the model to learn data-driven, semantically meaningful code assignments.
• Integrate the encoding schemes into Transformer-based models, enabling MoS to be applied efficiently in modern sequence-to-sequence architectures.

Experimental results

Research questions

• RQ1 Can BPE and Hybrid-LightRNN significantly reduce the memory and computation cost of Mixture of Softmaxes without sacrificing performance?
• RQ2 Does a learnable encoding scheme like Hybrid-LightRNN outperform heuristic methods like BPE in capturing semantic and syntactic structure in word representations?
• RQ3 To what extent does the ability to generate out-of-vocabulary (OOV) words contribute to the performance gain of BPE-MoS over Hybrid-LightRNN-MoS?
• RQ4 How does increasing the number of MoS components affect performance, and does the benefit exhibit diminishing returns?
• RQ5 Can the learned codebook in Hybrid-LightRNN group semantically or syntactically similar words into the same row or column, indicating meaningful structure learning?

Key findings

• BPE and Hybrid-LightRNN each reduce the time and memory consumption of MoS by approximately 50% on the IWSLT 2014 German-to-English translation task, with no performance degradation.
• On the WMT 2014 English-to-German translation benchmark, MoS with BPE or Hybrid-LightRNN achieves a BLEU score of 29.6, setting a new state-of-the-art result.
• The MoS model with BPE improves performance by 1.5 BLEU points on IWSLT 2014 German-to-English translation and 0.76 CIDEr points on image captioning compared to the baseline.
• Increasing the number of MoS components leads to improved performance, but with diminishing returns, indicating that a moderate number of mixtures (e.g., 3–5) is sufficient for high expressiveness.
• Hybrid-LightRNN outperforms both random and frequency-based codebook initialization by 2.68 and 1.55 BLEU points respectively, confirming the value of end-to-end learning of the encoding table.
• Even when OOV generation is disabled, BPE-MoS still outperforms Hybrid-LightRNN-MoS by 0.07 BLEU, indicating that BPE’s learned subword encoding captures data statistics more effectively than the learned table in Hybrid-LightRNN.