[Paper Review] Alternative Weighting Schemes for ELMo Embeddings

Nils Reimers, Iryna Gurevych|arXiv (Cornell University)|Apr 5, 2019

Electromagnetic Simulation and Numerical MethodsEngineering3 references21 citations

TL;DR

This paper evaluates alternative weighting schemes for ELMo embeddings, finding that learning a weighted average of only the first two biLM layers outperforms Peters et al.'s original learned weighted average across multiple NLP tasks. The proposed method improves performance on 7 out of 8 datasets, reduces training time by 19–44%, and avoids the performance degradation caused by the third (most abstract) layer.

ABSTRACT

ELMo embeddings (Peters et. al, 2018) had a huge impact on the NLP community and may recent publications use these embeddings to boost the performance for downstream NLP tasks. However, integration of ELMo embeddings in existent NLP architectures is not straightforward. In contrast to traditional word embeddings, like GloVe or word2vec embeddings, the bi-directional language model of ELMo produces three 1024 dimensional vectors per token in a sentence. Peters et al. proposed to learn a task-specific weighting of these three vectors for downstream tasks. However, this proposed weighting scheme is not feasible for certain tasks, and, as we will show, it does not necessarily yield optimal performance. We evaluate different methods that combine the three vectors from the language model in order to achieve the best possible performance in downstream NLP tasks. We notice that the third layer of the published language model often decreases the performance. By learning a weighted average of only the first two layers, we are able to improve the performance for many datasets. Due to the reduced complexity of the language model, we have a training speed-up of 19-44% for the downstream task.

Motivation & Objective

• To investigate whether the original learned weighted average of ELMo's three biLM layers is optimal for downstream NLP tasks.
• To evaluate alternative weighting schemes—individual layers, concatenation, fixed average, and variants of learned averaging—for ELMo embeddings.
• To determine if the third (last) layer of the biLM consistently contributes to performance or if it sometimes harms it.
• To develop a faster, more effective alternative to the original ELMo integration method that maintains or improves performance.
• To assess the impact of weighting schemes when ELMo is used alone versus in combination with other embeddings like GloVe.

Proposed method

• The authors evaluate five weighting schemes: using only the first, second, or third biLM layer; concatenating all three layers; computing a fixed average of all three; and learning a task-specific weighted average of all three layers.
• They introduce and evaluate a new scheme: learning a task-specific weighted average of only the first two biLM layers, excluding the third layer.
• The method uses a learnable, softmax-normalized weight vector s ∈ ℝ³ to compute a weighted average of the first two layer outputs, with a learnable scalar γ for scaling.
• The proposed method is tested on both a BiLSTM-CRF model using ELMo as the sole input and on AllenNLP models that combine ELMo with GloVe embeddings.
• Training and inference speed are measured to assess computational efficiency, particularly when the third layer is omitted.
• Experiments are conducted on eight benchmark NLP datasets: CoNLL-2003 NER, SNLI, SQuAD, SICK, SST-5, CoNLL-2005 parsing, CoNLL-2003 parsing, and GENIA.

Experimental results

Research questions

• RQ1 Does the original learned weighted average of all three ELMo layers yield the best performance across diverse NLP tasks?
• RQ2 Is the third (last) layer of the ELMo biLM consistently beneficial, or does it sometimes degrade performance?
• RQ3 Can a simpler weighting scheme—such as a learned average of only the first two layers—achieve better performance than the original method?
• RQ4 What is the impact of excluding the third biLM layer on both model performance and training speed?
• RQ5 How critical is the weighting scheme when ELMo is used as the sole input representation versus when combined with other embeddings?

Key findings

• Using only the second layer of the ELMo biLM outperforms the third layer on 7 out of 8 datasets, indicating that the most abstract representation is not always optimal.
• The learned weighted average of only the first two layers achieved higher performance than the original method on 7 out of 8 datasets, with statistically significant improvements on several.
• The proposed method reduced training time by 19–44% across models, with the largest speed-up observed in the SNLI model (44%) and the smallest in the parsing model (19%).
• For models using ELMo as the sole input, the weighting scheme had a significant impact; however, when combined with GloVe embeddings, performance differences between schemes were smaller and often not statistically significant.
• The third layer of the biLM was found to be harmful in many cases, and its exclusion improved performance, likely because the original learned weighting scheme forces the use of all three layers, including potentially detrimental ones.
• The fixed average and learned weighted average of all three layers showed negligible performance differences, suggesting that simple averaging is nearly as effective as learning complex weights.