[Paper Review] DeFINE: Deep Factorized Input Word Embeddings for Neural Sequence Modeling

Sachin Mehta, Rik Koncel-Kedziorski+2 more|arXiv (Cornell University)|Apr 30, 2020

Topic Modeling43 references13 citations

TL;DR

DeFINE introduces a deep factorized input word embedding method that reduces parameters and training time in neural sequence models by using a hierarchical architecture with skip-connections, enabling low-dimensional input and output layers. It achieves 6–20% lower perplexity than state-of-the-art methods and reduces parameters by up to 50% with minimal performance drop, improving models like Transformer-XL and AWD-LSTM.

ABSTRACT

For sequence models with large word-level vocabularies, a majority of network parameters lie in the input and output layers. In this work, we describe a new method, DeFINE, for learning deep word-level representations efficiently. Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods. DeFINE can be incorporated easily in new or existing sequence models. Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity. On WikiText-103, DeFINE reduces total parameters of Transformer-XL by half with minimal impact on performance. On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters, achieving comparable performance to state-of-the-art methods with fewer parameters. For machine translation, DeFINE improves a Transformer model by 2% while simultaneously reducing total parameters by 26%

Motivation & Objective

• To reduce the large number of parameters in input and output layers of sequence models with large vocabularies.
• To improve training efficiency without sacrificing model performance.
• To enable efficient integration into existing and new sequence models.
• To achieve state-of-the-art performance with fewer parameters through low-dimensional, factorized representations.
• To explore hierarchical factorization with skip-connections for deeper, more efficient word embedding learning.

Proposed method

• DeFINE employs a hierarchical architecture that factorizes word embeddings into low-dimensional components to reduce parameter count.
• It introduces novel skip-connections across layers to preserve gradient flow and improve training stability.
• The method uses a deep, factorized structure in input and output layers, replacing standard embedding matrices with shared, low-rank components.
• The architecture allows for efficient parameter sharing and reduces the total number of parameters in the network.
• DeFINE is designed to be plug-and-play, compatible with existing models like AWD-LSTM and Transformer-XL.
• The model uses standard training procedures but benefits from reduced parameter space and improved optimization dynamics.

Experimental results

Research questions

• RQ1 Can a hierarchical, factorized embedding architecture reduce model parameters without degrading performance?
• RQ2 How does DeFINE compare to adaptive input methods in terms of perplexity and parameter efficiency?
• RQ3 To what extent can DeFINE reduce training time and parameter count in Transformer-XL and AWD-LSTM?
• RQ4 Does DeFINE improve performance in machine translation tasks while reducing model size?
• RQ5 Can skip-connections in a factorized architecture enhance training stability and convergence?

Key findings

• On WikiText-103, DeFINE reduces the total parameters of Transformer-XL by 50% with minimal performance impact.
• On the Penn Treebank, DeFINE improves AWD-LSTM by 4 perplexity points while reducing parameters by 17%.
• DeFINE achieves performance comparable to state-of-the-art methods but with fewer parameters, demonstrating parameter efficiency.
• In machine translation, DeFINE improves a Transformer model by 2 BLEU points while reducing total parameters by 26%.
• DeFINE achieves a 6–20% relative reduction in perplexity compared to state-of-the-art methods across multiple benchmarks.
• The method maintains strong performance across diverse sequence modeling tasks, including language modeling and machine translation.