[Paper Review] Poincaré Embeddings for Learning Hierarchical Representations

Maximilian Nickel, Douwe Kiela|arXiv (Cornell University)|May 22, 2017

Advanced Graph Neural Networks28 references171 citations

TL;DR

The paper proposes learning embeddings in hyperbolic (Poincaré ball) space to capture both hierarchy and similarity, yielding superior performance on hierarchical data, networks, and lexical entailment compared to Euclidean embeddings. It introduces Riemannian optimization for training and demonstrates strong results on WordNet taxonomies, collaboration networks, and lexical entailment tasks.

ABSTRACT

Representation learning has become an invaluable approach for learning from symbolic data such as text and graphs. However, while complex symbolic datasets often exhibit a latent hierarchical structure, state-of-the-art methods typically learn embeddings in Euclidean vector spaces, which do not account for this property. For this purpose, we introduce a new approach for learning hierarchical representations of symbolic data by embedding them into hyperbolic space -- or more precisely into an n-dimensional Poincaré ball. Due to the underlying hyperbolic geometry, this allows us to learn parsimonious representations of symbolic data by simultaneously capturing hierarchy and similarity. We introduce an efficient algorithm to learn the embeddings based on Riemannian optimization and show experimentally that Poincaré embeddings outperform Euclidean embeddings significantly on data with latent hierarchies, both in terms of representation capacity and in terms of generalization ability.

Motivation & Objective

• Motivate learning representations that reflect latent hierarchies in symbolic data.
• Propose embedding objects in hyperbolic space (Poincaré ball) to capture hierarchy and similarity with fewer parameters.
• Develop an efficient Riemannian optimization algorithm to train embeddings under the Poincaré ball constraint.
• Demonstrate improved capacity and generalization on taxonomies, networks, and lexical entailment tasks.

Proposed method

• Embed symbols in the Poincaré ball Bd with distance d(u,v) = arcosh(1 + 2||u−v||^2 / ((1−||u||^2)(1−||v||^2))).
• Optimize a loss L(Θ) with Euclidean gradients adjusted by the Poincaré metric using Riemannian gradient and retraction (Rθ(v) = θ + v).
• Constrain embeddings to lie inside the ball with projection proj(θ) = θ/||θ|| − ε if ||θ|| ≥ 1 (ε = 1e-5).
• Use stochastic/mini-batch Riemannian optimization (RSGD/RSVRG) and a burn-in phase with reduced learning rate to stabilize angular layout.
• Train with a negative sampling objective for observed hierarchies (soft ranking) and compare against Euclidean and translational baselines.
• Evaluate on WordNet noun hierarchy reconstruction and link prediction, large network embeddings, and lexical entailment via a graded metric.

Experimental results

Research questions

• RQ1 Can hyperbolic geometry provide more parsimonious, hierarchical embeddings than Euclidean geometry for data with latent hierarchies?
• RQ2 Do Poincaré embeddings improve representation capacity and generalization in taxonomies and networks due to their negative curvature bias?
• RQ3 Are Poincaré embeddings effective for lexical entailment tasks that require modeling hierarchy within language?
• RQ4 How does Riemannian optimization perform for training these hyperbolic embeddings at scale?

Key findings

• Poincaré embeddings achieve higher MAP and lower mean rank than Euclidean and Translational methods in WordNet reconstruction and link prediction, at comparable or lower dimensionality.
• In network link prediction, Poincaré embeddings outperform Euclidean embeddings across four networks, especially in low dimensions (e.g., AstroPh, CondMat, GrQc, HepPh).
• Poincaré embeddings yield state-of-the-art performance on lexical entailment tasks (HyperLex) when assessing graded is-a relationships, outperforming several WordNet-based baselines.
• Hyperbolic geometry enables strong hierarchical representation with fewer parameters, improving generalization and robustness to missing data.
• Optimization via Riemannian methods with the Poincaré ball model scales to large datasets and can be parallelized effectively (e.g., Hogwild).