[Paper Review] Hash Layers For Large Sparse Models

Stephen Roller, Sainbayar Sukhbaatar+2 more|arXiv (Cornell University)|Jun 8, 2021

Topic Modeling36 references48 citations

TL;DR

Hash Layers route tokens to fixed expert FFNs via precomputed hashes, achieving competitive or better perplexity than Switch Transformers and BASE Layers without routing parameters or extra objectives.

ABSTRACT

We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforward layer to hash to different sets of weights depending on the current token, over all tokens in the sequence. We show that this procedure either outperforms or is competitive with learning-to-route mixture-of-expert methods such as Switch Transformers and BASE Layers, while requiring no routing parameters or extra terms in the objective function such as a load balancing loss, and no sophisticated assignment algorithm. We study the performance of different hashing techniques, hash sizes and input features, and show that balanced and random hashes focused on the most local features work best, compared to either learning clusters or using longer-range context. We show our approach works well both on large language modeling and dialogue tasks, and on downstream fine-tuning tasks.

Motivation & Objective

• Motivate scalable sparse routing in Transformers without learnable routing parameters.
• Propose a hash-based routing mechanism that maps tokens to fixed expert modules.
• Evaluate hashing strategies and multi-hash extensions across language modeling and dialogue tasks.
• Compare performance and efficiency against Switch Transformers and BASE Layers.
• Analyze how hash choices affect training dynamics and downstream fine-tuning.

Proposed method

• Replace FFN in a Transformer layer with an MoE-style set of expert FFNs accessed via a fixed hash-based routing function.
• Fix the hash function in advance (no learned routing parameters) and route tokens to a single expert per token.
• Explore multiple hash strategies (Random, Balanced, Bigram, Previous Token, Position-based, Oracle/Future-based, Clustering, Dispersed) and their impact on performance.
• Introduce MultiHash Layers that compose several hash functions to select sub-parameters and concatenate their outputs.
• Conduct experiments on Pushshift Reddit, RoBERTa+cc100en, Wikitext-103, and BST dialogue tasks, comparing against dense baselines, Switch Transformers, and BASE Layers.
• Provide ablations on hash function choice, layer position, and multi-hashing effects.

Experimental results

Research questions

• RQ1 Can a fixed, hash-based routing scheme achieve competitive performance to learnable routing in sparse Transformer models?
• RQ2 Which hash strategies (random, balanced, token-based, n-gram, clustering-based, oracle-based) yield the best trade-offs between perplexity and training efficiency?
• RQ3 Does placing hash-based routing in later Transformer layers improve performance compared to earlier layers?
• RQ4 What is the impact of multi-hash routing on model performance with fixed compute budgets?
• RQ5 How do Hash Layers compare to BASE Layers in terms of accuracy and training/update efficiency at scale?

Key findings

• Hash Layers outperform or match Switch Transformers on Reddit and RoBERTa+cc100en with similar parameter counts.
• Hash Layers with 64 or 128 modules achieve lower perplexity than their Switch counterparts on the same compute budget.
• Hash routing without learned gating is robust, simple to implement, and avoids load-balancing losses and routing optimization difficulties.
• Later-layer Hash Layer placement yields better perplexity than earlier layers, aligning with findings for BASE Layers.
• Multi-hashing (2–8 hashes) improves performance over a single hash while keeping total parameters constant.
• Compared to BASE Layers, Hash Layers give better or equal perplexity and higher updates-per-second due to avoiding de-correlation steps in routing.