[Paper Review] Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

Yi Tay, Vinh Q. Tran+8 more|arXiv (Cornell University)|Jun 23, 2021

Natural Language Processing Techniques62 references83 citations

TL;DR

Charformer introduces a gradient-based subword tokenization (GBST) module that learns latent subword representations from bytes and integrates it into a Transformer, achieving competitive performance with improved speed versus byte-level and subword baselines.

ABSTRACT

State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings. In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. To this end, we introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level. Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28%-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end.

Motivation & Objective

• Motivate tokenization as a learnable, end-to-end component rather than a fixed pre-processing step.
• Balance the flexibility of character-level modeling with subword efficiency via GBST.
• Develop Charformer to operate on byte-level inputs while maintaining competitive downstream performance.
• Demonstrate speed and memory efficiency advantages over existing byte-level and subword models.
• Provide interpretable insights by visualizing learned latent subwords.

Proposed method

• Propose Gradient-Based Subword Tokenization (GBST) that enumerates candidate blocks of up to size M and learns a block-scoring network to assign probabilities for each block at each position.
• Form latent subword representations by a soft weighted sum of candidate blocks, enabling differentiable, end-to-end learning of subword segmentation.
• Downsample the latent subword sequence with a fixed pooling operation to reduce sequence length before feeding into a Transformer stack.
• Use a Transformer encoder-decoder that operates on the downsampled latent subwords, with a scalable variant (SBase) that allocates more capacity to the encoder.
• Train with a span-masking pre-training scheme similar to T5, adapting it to the byte-level setup.
• Explore architectural variants (e.g., pre-GBST convolutions, block score calibration, downsampling rates) to assess efficiency and performance.

Experimental results

Research questions

• RQ1 Can GBST learn meaningful latent subword representations directly from character/byte sequences?
• RQ2 How does Charformer perform compared to subword-based and other tokenization-free models across monolingual and multilingual tasks?
• RQ3 What are the speed, memory, and parameter efficiency benefits of Charformer relative to existing baselines?
• RQ4 Does re-scaling the Transformer encoder (SBase) yield advantages for character/byte-level models?
• RQ5 Are the learned latent subwords interpretable and robust to noisy text?

Key findings

• Charformer outperforms strong character-level baselines at the same parameter count across English GLUE tasks and long-document classification.
• Charformer matches or surpasses subword-based models (e.g., BERT, T5) on several benchmarks, with the SBase variant often performing best.
• Charformer is faster and more memory-efficient than comparable byte-level and subword-based models, e.g., Byte-level T5 and T5-based variants, and training/inference costs scale favorably.
• The SBase configuration, with a deeper encoder and smaller parameter footprint, can outperform non-scaled baselines while using roughly 40–50% fewer parameters and achieving 2x–10x speed improvements in some settings.
• In multilingual settings, Charformer SBase is competitive with subword baselines and LongPT variants, with strong in-language and translate-train performance; zero-shot cross-lingual transfer remains challenging for token-free models.