[Paper Review] Learning Numeral Embeddings

Chengyue Jiang, Zhonglin Nian+5 more|arXiv (Cornell University)|Dec 28, 2019

Mathematics Education and Teaching Techniques30 references4 citations

TL;DR

This paper proposes two novel numeral embedding methods that represent any numeral as a weighted average of learned prototype numeral embeddings, using self-organizing maps or Gaussian mixture models to induce prototypes. The approach effectively handles out-of-vocabulary numerals and improves performance on numeral-related NLP tasks, outperforming baselines in word similarity, numeracy, prediction, and sequence labeling, especially in low-resource settings.

ABSTRACT

Word embedding is an essential building block for deep learning methods for natural language processing. Although word embedding has been extensively studied over the years, the problem of how to effectively embed numerals, a special subset of words, is still underexplored. Existing word embedding methods do not learn numeral embeddings well because there are an infinite number of numerals and their individual appearances in training corpora are highly scarce. In this paper, we propose two novel numeral embedding methods that can handle the out-of-vocabulary (OOV) problem for numerals. We first induce a finite set of prototype numerals using either a self-organizing map or a Gaussian mixture model. We then represent the embedding of a numeral as a weighted average of the prototype number embeddings. Numeral embeddings represented in this manner can be plugged into existing word embedding learning approaches such as skip-gram for training. We evaluated our methods and showed its effectiveness on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling.

Motivation & Objective

• To address the poor handling of numerals in existing word embedding methods due to their infinite variety and low frequency in training data.
• To develop a method that can generate meaningful embeddings for out-of-vocabulary (OOV) numerals, including rare or unseen numbers.
• To integrate numeral embeddings into standard word embedding frameworks like skip-gram without requiring model retraining from scratch.
• To evaluate the effectiveness of the proposed method on both intrinsic and extrinsic tasks involving numerals.
• To improve downstream NLP performance in scenarios where numeral meaning and magnitude are critical, such as in customer service or clinical text.

Proposed method

• Induce a finite set of prototype numerals from training data using either a self-organizing map (SOM) or a Gaussian mixture model (GMM).
• Represent the embedding of any target numeral as a weighted average of the prototype embeddings, where weights are based on the numerical distance between the target and each prototype.
• Use the prototype-based numeral embeddings as input to standard word embedding models such as skip-gram for joint training with non-numerical words.
• Train the model end-to-end using standard objective functions, allowing the system to learn both word and numeral representations simultaneously.
• Apply the learned embeddings to downstream tasks such as sequence labeling, word similarity, and numeral prediction.
• Use a validation set for model selection and avoid fine-tuning or additional features like part-of-speech or character-level embeddings in evaluation.

Experimental results

Research questions

• RQ1 Can prototype-based numeral embeddings effectively represent out-of-vocabulary numerals that are rare or unseen in training data?
• RQ2 How well do the proposed numeral embeddings perform on intrinsic tasks like word similarity and embedding numeracy compared to baseline methods?
• RQ3 To what extent do the proposed embeddings improve performance on extrinsic tasks such as numeral prediction and sequence labeling?
• RQ4 How does the method generalize under low-resource conditions with only 10% or 30% of the training data?
• RQ5 Can the method distinguish between different semantic senses of numerals, such as a numeral representing a year versus a quantity?

Key findings

• The proposed method consistently outperforms all baselines on sequence labeling tasks, achieving the highest F1 scores across original, augmented, and hard test sets.
• On the customer service dataset, the method achieved an F1 score of 93.24 on the original test set and 92.10 on the hard test set when using 100% training data, significantly outperforming NumAsTok.
• In low-resource settings with only 10% of the training data, the proposed method showed a larger performance advantage over baselines, indicating better generalization and lower data dependency.
• The method demonstrated robustness to perturbations in numerals, with minimal performance drop on the augmented test set, unlike NumAsTok, which showed a significant decline due to OOV issues.
• The prototype-based approach achieved high accuracy in embedding numeracy, indicating that the method successfully encodes magnitude and numerical relationships.
• The results suggest that the method can effectively model both the semantic and quantitative aspects of numerals, enabling better reasoning in NLP systems.