[Paper Review] Self-organized Hierarchical Softmax

Yikang Shen, Shawn Tan+2 more|arXiv (Cornell University)|Jul 26, 2017

Topic Modeling24 references3 citations

TL;DR

This paper proposes a self-organized hierarchical softmax that learns word clusters during language model training based on contextual similarity, rather than relying on predefined structures like frequency or word embeddings. The method achieves full softmax-level performance in language modeling and superior results in sentence compression, while being over three times faster than standard softmax due to context-driven, semantically meaningful clustering.

ABSTRACT

We propose a new self-organizing hierarchical softmax formulation for neural-network-based language models over large vocabularies. Instead of using a predefined hierarchical structure, our approach is capable of learning word clusters with clear syntactical and semantic meaning during the language model training process. We provide experiments on standard benchmarks for language modeling and sentence compression tasks. We find that this approach is as fast as other efficient softmax approximations, while achieving comparable or even better performance relative to similar full softmax models.

Motivation & Objective

• To address the computational inefficiency of full softmax in large-vocabulary language models.
• To improve model performance by learning hierarchical word structures that reflect syntactic and semantic relationships.
• To develop a hierarchical softmax that dynamically organizes words into clusters based on contextual prediction during training.
• To achieve faster inference and training speeds comparable to approximate softmax methods while maintaining high accuracy.
• To demonstrate that self-organized clusters can implicitly capture grammatical roles and semantic similarity without explicit POS tagging.

Proposed method

• The model learns a hierarchical tree structure during language model training, where word clusters are formed based on their predictive context rather than static word frequencies or embeddings.
• A clustering objective is introduced to maximize the likelihood of a word belonging to its assigned cluster given its preceding context.
• The hierarchical softmax computes word probabilities as the product of node probabilities along the path from root to leaf, reducing computation from O(|V|) to O(log|V|).
• The algorithm uses backpropagation to jointly optimize the language model and the hierarchical structure, enabling end-to-end learning of meaningful clusters.
• Word clusters are formed dynamically through gradient-based optimization, favoring words with similar contextual behavior.
• The method is applied within an LSTM-based language model and a sequence-to-sequence model with attention for sentence compression.

Experimental results

Research questions

• RQ1 Can a hierarchical softmax structure be learned end-to-end during language model training to improve performance?
• RQ2 Does context-driven clustering of words lead to semantically and syntactically meaningful word groups?
• RQ3 Can a self-organized hierarchical softmax match or exceed the performance of full softmax while maintaining computational efficiency?
• RQ4 How does the learned structure compare to predefined or embedding-based clustering in terms of semantic coherence?
• RQ5 To what extent does the hierarchical structure improve downstream tasks like sentence compression?

Key findings

• The self-organized hierarchical softmax achieved a sentence compression F1 score of 31.95 on the Gigaword dataset, outperforming the full softmax (31.52) and state-of-the-art methods.
• On the language modeling task, the method matched the performance of full softmax while being over three times faster, with training time reduced from 210 minutes to 63 minutes on the Gigaword5 set.
• Word clusters learned by the model showed strong semantic and syntactic coherence, grouping words by grammatical role and meaning without explicit part-of-speech supervision.
• The model produced more accurate and fluent summaries than full softmax, as shown in qualitative examples from the Gigaword test set.
• The approach demonstrated that context-aware clustering during training leads to better generalization and efficiency in sequence modeling tasks.
• The method achieved a 3× speedup over standard softmax while maintaining or improving performance, confirming its practical efficiency.