[Paper Review] Probing BERT in Hyperbolic Spaces

Boli Chen, Yao Fu+5 more|arXiv (Cornell University)|Apr 8, 2021

Topic ModelingComputer Science20 references19 citations

TL;DR

This paper introduces a Poincaré probe to analyze BERT's contextualized embeddings in hyperbolic space, leveraging the Poincaré ball model's natural tree-like geometry to better capture hierarchical linguistic structures. The method demonstrates superior performance in recovering syntactic dependency trees and sentiment hierarchies compared to Euclidean probes, especially for deeper trees and longer sentences, suggesting BERT's syntactic representations may inherently favor non-Euclidean geometry.

ABSTRACT

Recently, a variety of probing tasks are proposed to discover linguistic properties learned in contextualized word embeddings. Many of these works implicitly assume these embeddings lay in certain metric spaces, typically the Euclidean space. This work considers a family of geometrically special spaces, the hyperbolic spaces, that exhibit better inductive biases for hierarchical structures and may better reveal linguistic hierarchies encoded in contextualized representations. We introduce a Poincare probe, a structural probe projecting these embeddings into a Poincare subspace with explicitly defined hierarchies. We focus on two probing objectives: (a) dependency trees where the hierarchy is defined as head-dependent structures; (b) lexical sentiments where the hierarchy is defined as the polarity of words (positivity and negativity). We argue that a key desideratum of a probe is its sensitivity to the existence of linguistic structures. We apply our probes on BERT, a typical contextualized embedding model. In a syntactic subspace, our probe better recovers tree structures than Euclidean probes, revealing the possibility that the geometry of BERT syntax may not necessarily be Euclidean. In a sentiment subspace, we reveal two possible meta-embeddings for positive and negative sentiments and show how lexically-controlled contextualization would change the geometric localization of embeddings. We demonstrate the findings with our Poincare probe via extensive experiments and visualization. Our results can be reproduced at https://github.com/FranxYao/PoincareProbe.

Motivation & Objective

• To investigate whether BERT's syntactic and sentiment representations are better modeled in hyperbolic space than in Euclidean space.
• To develop a Poincaré probe that explicitly encodes hierarchical structures using the geometry of hyperbolic space.
• To evaluate whether the probe's sensitivity to linguistic hierarchies is higher than that of standard Euclidean probes.
• To explore the geometric localization of sentiment embeddings and the impact of lexically controlled contextualization.

Proposed method

• The Poincaré probe projects BERT's contextualized word embeddings into a Poincaré ball, a model of hyperbolic space with constant negative curvature.
• The probe uses Möbius operations—addition, matrix multiplication, exponential, and logarithmic maps—defined for the generalized Poincaré ball model to maintain geometric consistency.
• For syntax probing, the hierarchy is defined by dependency tree depth, with nodes closer to the origin representing higher syntactic levels.
• For sentiment probing, two meta-embeddings ([POS] and [NEG]) are placed at the origin, and word sentiment polarity determines their radial distance.
• The probe is trained to predict syntactic structure or sentiment polarity using geometric relationships in the Poincaré space.
• The method is evaluated using UUAS (Unlabeled Ullmann's Accuracy Score) and DSpr (Dependency Structure Precision) on syntactic and sentiment tasks.

Experimental results

Research questions

• RQ1 Does hyperbolic geometry better capture the hierarchical structure of BERT's syntactic representations than Euclidean geometry?
• RQ2 Can a Poincaré probe recover dependency trees more accurately than a Euclidean probe, especially for deeper or longer syntactic structures?
• RQ3 How do sentiment embeddings in BERT localize geometrically in hyperbolic space, and what role does lexically controlled contextualization play?
• RQ4 Is the Poincaré probe sensitive to linguistic structure without being a strong parser in itself?

Key findings

• The Poincaré probe achieved a UUAS of 84.5 on the BERTbase10 syntactic probing task, significantly outperforming the best Euclidean probe (84.0) with non-linearity, indicating higher sensitivity to syntactic structure.
• For deeper trees and longer sentences, the Poincaré probe showed substantially higher parsing scores than the Euclidean probe, suggesting BERT's syntactic geometry may not be Euclidean.
• In sentiment probing, the Poincaré probe revealed two distinct meta-embeddings for positive and negative sentiments, with stronger sentiment words localized closer to their respective meta-embeddings.
• The probe maintained low scores on random baseline embeddings (UUAS ~19.9), confirming it does not act as a powerful parser but is sensitive only to intrinsic linguistic structure.
• Experiments with varying curvature showed that curvature values of -0.5 and -1.0 yielded the highest DSpr (0.70) and UUAS (40.5) on the distance task, indicating optimal geometric fit for syntactic hierarchies.
• Visualization confirmed that hyperbolic space better accommodates tree structures due to its exponentially growing volume with radius, unlike the polynomial growth in Euclidean space.