[Paper Review] Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings

Matt Le, Stephen Roller+3 more|arXiv (Cornell University)|Jan 1, 2019

Topic Modeling55 references21 citations

TL;DR

This paper proposes a novel method for inferring concept hierarchies from text corpora by combining Hearst patterns with hyperbolic embeddings. It leverages the hierarchical geometry of hyperbolic space to improve transitivity, efficiency, and interpretability, achieving state-of-the-art performance on multiple hypernymy benchmarks while reducing embedding dimensionality by over an order of magnitude compared to SVD-based methods.

ABSTRACT

We consider the task of inferring is-a relationships from large text corpora. For this purpose, we propose a new method combining hyperbolic embeddings and Hearst patterns. This approach allows us to set appropriate constraints for inferring concept hierarchies from distributional contexts while also being able to predict missing is-a relationships and to correct wrong extractions. Moreover -- and in contrast with other methods -- the hierarchical nature of hyperbolic space allows us to learn highly efficient representations and to improve the taxonomic consistency of the inferred hierarchies. Experimentally, we show that our approach achieves state-of-the-art performance on several commonly-used benchmarks.

Motivation & Objective

• To address the challenge of inferring accurate, consistent, and efficient concept hierarchies from large-scale text corpora.
• To overcome limitations of pattern-based methods like Hearst patterns, which suffer from sparsity and false negatives.
• To improve upon distributional methods by incorporating hierarchical inductive bias via hyperbolic geometry.
• To enable low-dimensional, yet highly effective, representations of taxonomic structures in natural language.

Proposed method

• Extract potential is-a relationships using Hearst patterns from a large text corpus (GigaWord + Wikipedia), forming a directed, weighted Hearst graph.
• Embed the resulting graph into hyperbolic space using the Lorentz model, leveraging its intrinsic hierarchical structure.
• Introduce a new method to compute hyperbolic entailment cones in the Lorentz model to enforce transitivity and improve optimization.
• Use the norm of embeddings to capture term generality and distance to capture semantic similarity.
• Apply the model to predict missing hypernymy relations and correct erroneous extractions via geometric constraints.
• Optimize the joint embedding to best explain the observed Hearst pattern extractions while maintaining taxonomic consistency.

Experimental results

Research questions

• RQ1 Can hyperbolic embeddings improve the consistency and accuracy of inferred concept hierarchies compared to Euclidean or SVD-based methods?
• RQ2 To what extent can hyperbolic geometry reduce the required embedding dimensionality while preserving or improving performance?
• RQ3 How effective is the proposed method in recovering missing and transitive hypernymy relations in a taxonomy?
• RQ4 Can the combination of Hearst patterns and hyperbolic embeddings outperform existing state-of-the-art methods on standard hypernymy benchmarks?
• RQ5 Does the geometric structure of hyperbolic space enhance interpretability by encoding both similarity and hierarchy in a single vector space?

Key findings

• The proposed HYPECONES method achieves state-of-the-art performance on seven out of nine hypernymy benchmarks, including BLESS, LEDS, EVAL, SHWARTZ, WBLESS, BIBLESS, and HYPERLEX.
• On the HYPERLEX dataset, the method achieves a Spearman correlation of 0.59, significantly outperforming prior Poincaré GLOVE (0.341) and matching or exceeding SVD-based baselines.
• On the WBLESS benchmark, the method achieves 90.9% accuracy, a substantial improvement over prior methods, demonstrating strong performance on detection tasks.
• The method reduces required embedding dimensionality from 300D (SVD) to just 20D while maintaining or improving performance, highlighting its efficiency.
• The ablation study confirms that hyperbolic entailment cones improve transitivity, correctly inferring missing transitive relations like (male horse, is-a, equine) even when the direct path is absent in the Hearst graph.
• The reconstruction of WORDNET subtrees (animals, plants, vehicles) shows a relative improvement of over 40% compared to the best non-hyperbolic baseline, validating the method’s ability to recover full taxonomies.