[Paper Review] BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages

Benjamin Heinzerling, Michael Strube|arXiv (Cornell University)|Oct 5, 2017

Topic Modeling14 references128 citations

TL;DR

BPEmb provides pre-trained subword embeddings for 275 languages using Byte-Pair Encoding, enabling tokenization-free representations that are competitive with FastText and other subword methods while using far fewer resources.

ABSTRACT

We present BPEmb, a collection of pre-trained subword unit embeddings in 275 languages, based on Byte-Pair Encoding (BPE). In an evaluation using fine-grained entity typing as testbed, BPEmb performs competitively, and for some languages bet- ter than alternative subword approaches, while requiring vastly fewer resources and no tokenization. BPEmb is available at https://github.com/bheinzerling/bpemb

Motivation & Objective

• Publish a large collection of pre-trained BPE-based subword embeddings across 275 languages.
• Evaluate BPEmb on a fine-grained entity typing task and compare to FastText and character embeddings.
• Demonstrate tokenization-free representations and assess resource efficiency versus alternatives.
• Analyze the impact of BPE merge operations and embedding dimensionality on performance.

Proposed method

• Apply Byte-Pair Encoding (BPE) to all Wikipedias to obtain BPE symbol inventories at multiple merge counts.
• Pre-train subword embeddings for BPE symbols using GloVe on Wikipedia-derived text.
• Provide embeddings for various BPE merge operation counts (1k–200k) and embedding dimensions (25–300).
• Evaluate subword representations on a fine-grained entity typing task using mentions from Wikidata mapped to Freebase types.
• Compare BPEmb against FastText (tokenized and subword-aware) and character-based embeddings across architectures (RNN, CNN, and averaging).
• Report performance distributions over extensive hyper-parameter searches and report language- and architecture-specific results.

Experimental results

Research questions

• RQ1 Can BPEmb achieve competitive or superior performance to FastText and character embeddings on fine-grained entity typing across multiple languages?
• RQ2 Do tokenization-free BPEmb representations match tokenized methods in accuracy while offering substantial resource savings?
• RQ3 How do the number of BPE merge operations and embedding dimensionality affect performance across languages and architectures?
• RQ4 Which architecture (RNN, CNN, or averaging) best utilizes BPEmb subword representations for entity typing?

Key findings

• BPEmb outperforms all other subword units across architectures for the English dataset (BPEmb mean 0.624, std 0.029; max 0.65) and can match or exceed FastText performance while using far less memory.
• BPEmb can achieve competitive results with low embedding dimensionality (e.g., 11 MB for 100k BPE symbols at 25 dimensions, versus 6 GB for FastText with 3 million embeddings at 300 dimensions).
• In multilingual experiments, high-resource languages without explicit tokenization achieve comparable results to FastText when tokenization is not used; results vary for medium- to low-resource languages with BPEmb giving gains on Tibetan and Lao, and a degradation on Khmer due to unicode handling differences.
• Table-like language results show BPEmb matching or slightly improving FastText across several languages (e.g., English 65.4 vs 62.9; Chinese 72.0 vs 71.0; Japanese 61.4 vs 62.3).
• Averaging embeddings is the weakest architecture; RNNs perform slightly better than CNNs, though with higher training time; FastText shows lower variance across hyper-parameters, while BPEmb and character models exhibit higher variance.