[Paper Review] Multi hash embeddings in spaCy

Lester James V. Miranda, Ákos Kádár+4 more|arXiv (Cornell University)|Dec 19, 2022

Topic Modeling4 citations

TL;DR

This paper introduces MultiHashEmbed, a memory-efficient word embedding method in spaCy that uses multiple hash functions and orthographic features (normalization, prefix, suffix, shape) to generate dense vector representations without storing individual word vectors. Experiments show it achieves competitive NER performance with significantly reduced memory usage, though multiple hash functions provide minimal gains except on large datasets like OntoNotes, where three functions improved F1 by 5%.

ABSTRACT

The distributed representation of symbols is one of the key technologies in machine learning systems today, playing a pivotal role in modern natural language processing. Traditional word embeddings associate a separate vector with each word. While this approach is simple and leads to good performance, it requires a lot of memory for representing a large vocabulary. To reduce the memory footprint, the default embedding layer in spaCy is a hash embeddings layer. It is a stochastic approximation of traditional embeddings that provides unique vectors for a large number of words without explicitly storing a separate vector for each of them. To be able to compute meaningful representations for both known and unknown words, hash embeddings represent each word as a summary of the normalized word form, subword information and word shape. Together, these features produce a multi-embedding of a word. In this technical report we lay out a bit of history and introduce the embedding methods in spaCy in detail. Second, we critically evaluate the hash embedding architecture with multi-embeddings on Named Entity Recognition datasets from a variety of domains and languages. The experiments validate most key design choices behind spaCy's embedders, but we also uncover a few surprising results.

Motivation & Objective

• To design a memory-efficient alternative to traditional word embeddings that scales to large vocabularies without explicit storage of word vectors.
• To evaluate the effectiveness of spaCy’s MultiHashEmbed architecture across diverse NER datasets and languages.
• To identify key hyperparameters—such as number of hash functions and feature types—that influence performance and memory usage.
• To provide empirical guidance for users on customizing spaCy’s default NER pipelines for better efficiency and accuracy.
• To assess whether additional orthographic features improve generalization, especially for out-of-vocabulary or rare words.

Proposed method

• MultiHashEmbed uses the hashing trick to map words to fixed-size vector representations without storing a separate vector per word, reducing memory footprint.
• Each word is encoded using a combination of normalized form, prefix, suffix, and word shape features, which are independently hashed into embedding space.
• The system employs multiple hash functions (1–4) to reduce collision risks and improve vector uniqueness across the vocabulary.
• Embedding vectors are learned end-to-end within spaCy’s Thinc-based machine learning framework, enabling fine-tuning and integration with downstream NER models.
• The method combines subword-level representations (inspired by fastText) with orthographic patterns to improve robustness to spelling variations and OOV words.
• Experiments compare MultiHashEmbed against standard dense embeddings on multiple NER datasets, measuring F1 scores and memory efficiency across varying configurations.

Experimental results

Research questions

• RQ1 How does MultiHashEmbed compare in performance to traditional dense word embeddings in Named Entity Recognition tasks across diverse domains and languages?
• RQ2 What is the impact of using multiple hash functions on the accuracy and collision rate of the embedding layer?
• RQ3 To what extent do orthographic features (normalization, prefix, suffix, shape) improve performance, especially on rare or out-of-vocabulary words?
• RQ4 How do hyperparameters such as the number of hash rows and number of hash functions affect model accuracy and memory usage?
• RQ5 Does the inclusion of subword and morphological features lead to better generalization, or do they degrade performance on seen entities?

Key findings

• MultiHashEmbed achieves competitive NER performance compared to traditional embeddings while using significantly less memory, validating its design for efficiency.
• Using three hash functions instead of one improved F1 by 5% on the large OntoNotes dataset, but no further gains were observed with four functions.
• The default setting of 2500 hash rows was found to be sufficient for most datasets, with no significant benefit from increasing the number of rows beyond this threshold.
• Orthographic features improved overall performance on some datasets (e.g., AnEM), but unexpectedly degraded F1 scores on seen entities, suggesting potential overfitting or feature interference.
• Multiple hash functions provided minimal performance gains on smaller datasets like CoNLL and WNUT2017, likely due to the mitigating effect of other orthographic features on collision risk.
• The study recommends that users experiment with reducing the number of hash rows and using a single hash function to reduce memory and improve inference efficiency without major accuracy loss.