[Paper Review] DyERNIE: Dynamic Evolution of Riemannian Manifold Embeddings for Temporal Knowledge Graph Completion

Zhen Han, Yunpu Ma+2 more|arXiv (Cornell University)|Nov 8, 2020

Advanced Graph Neural Networks41 references4 citations

TL;DR

DyERNIE proposes a non-Euclidean approach for temporal knowledge graph completion by modeling evolving entity representations on a product of Riemannian manifolds with varying curvatures. It uses velocity vectors in tangent spaces to capture temporal dynamics, achieving state-of-the-art performance on three benchmark datasets by better modeling hierarchical and cyclic structures inherent in temporal KGs.

ABSTRACT

There has recently been increasing interest in learning representations of temporal knowledge graphs (KGs), which record the dynamic relationships between entities over time. Temporal KGs often exhibit multiple simultaneous non-Euclidean structures, such as hierarchical and cyclic structures. However, existing embedding approaches for temporal KGs typically learn entity representations and their dynamic evolution in the Euclidean space, which might not capture such intrinsic structures very well. To this end, we propose Dy- ERNIE, a non-Euclidean embedding approach that learns evolving entity representations in a product of Riemannian manifolds, where the composed spaces are estimated from the sectional curvatures of underlying data. Product manifolds enable our approach to better reflect a wide variety of geometric structures on temporal KGs. Besides, to capture the evolutionary dynamics of temporal KGs, we let the entity representations evolve according to a velocity vector defined in the tangent space at each timestamp. We analyze in detail the contribution of geometric spaces to representation learning of temporal KGs and evaluate our model on temporal knowledge graph completion tasks. Extensive experiments on three real-world datasets demonstrate significantly improved performance, indicating that the dynamics of multi-relational graph data can be more properly modeled by the evolution of embeddings on Riemannian manifolds.

Motivation & Objective

• To address the limitation of Euclidean space in capturing intrinsic geometric structures like hierarchy and cycles in temporal knowledge graphs.
• To model the dynamic evolution of entities over time in a geometrically principled way beyond static or linear representations.
• To develop a method that learns both stationary structural dependencies and time-varying behaviors through Riemannian manifold dynamics.
• To demonstrate that product manifolds with heterogeneous curvatures better reflect complex geometric patterns in real-world temporal KGs.

Proposed method

• Entity representations are initialized on a product of Riemannian manifolds with different curvatures, estimated from the sectional curvatures of the underlying temporal KG data.
• Temporal evolution is modeled via velocity vectors defined in the tangent space at each entity's initial embedding, enabling smooth geodesic motion on the manifold.
• The time-dependent embedding is generated by exponentiating the initial embedding with the velocity vector scaled by time, using Riemannian exponential maps.
• The model uses a predicate-aware distance function that adjusts subject and object embeddings based on relation-specific parameters.
• The curvature and dimensionality of component manifolds are selected based on empirical analysis of sectional curvatures and data geometry.
• Ablation studies confirm the importance of dynamic components and the superiority of linear evolution over complex periodic functions.

Experimental results

Research questions

• RQ1 Can modeling temporal knowledge graph dynamics on Riemannian manifolds with heterogeneous curvatures improve link prediction performance compared to Euclidean baselines?
• RQ2 How do velocity vectors in tangent spaces effectively capture the time-varying behavior of entities in multi-relational temporal graphs?
• RQ3 To what extent do product manifolds with varying curvatures reflect the intrinsic geometric structures—such as hierarchy and cycles—present in real-world temporal KGs?
• RQ4 Does the dynamic evolution of embeddings on Riemannian manifolds outperform static or purely periodic representation functions in modeling temporal KGs?

Key findings

• DyERNIE achieves a filtered MRR of 55.87 on ICEWS14 using linear evolution, significantly outperforming non-linear periodic and combined models.
• The ablation study shows that removing the dynamic component reduces Hits@1 to about half, proving the critical role of temporal evolution.
• Entities with high degrees in the graph are consistently located near the origin of the Poincaré ball, indicating hierarchical structure capture.
• Velocity norms are negatively correlated with node degrees, suggesting stable, low-movement dynamics for central, high-degree entities.
• The model’s performance degrades substantially when the distance function is simplified, especially when removing the translation vector.
• The product manifold structure enables better modeling of complex geometric patterns, as evidenced by improved performance across all three benchmark datasets.