[Paper Review] Rot-Pro: Modeling Transitivity by Projection in Knowledge Graph Embedding

Tengwei Song, Jie Luo+1 more|arXiv (Cornell University)|Oct 27, 2021

Advanced Graph Neural Networks27 references21 citations

TL;DR

Rot-Pro proposes a knowledge graph embedding model that combines relational rotation with projection to explicitly model transitivity, a relation pattern previously underexplored in existing models. By theoretically proving that transitivity can be captured via idempotent projections and integrating this with RotatE's rotational mechanism, Rot-Pro achieves state-of-the-art performance on link prediction in transitive datasets like Countries, outperforming prior models in YAGO3-10 and FB15k-237.

ABSTRACT

Knowledge graph embedding models learn the representations of entities and relations in the knowledge graphs for predicting missing links (relations) between entities. Their effectiveness are deeply affected by the ability of modeling and inferring different relation patterns such as symmetry, asymmetry, inversion, composition and transitivity. Although existing models are already able to model many of these relations patterns, transitivity, a very common relation pattern, is still not been fully supported. In this paper, we first theoretically show that the transitive relations can be modeled with projections. We then propose the Rot-Pro model which combines the projection and relational rotation together. We prove that Rot-Pro can infer all the above relation patterns. Experimental results show that the proposed Rot-Pro model effectively learns the transitivity pattern and achieves the state-of-the-art results on the link prediction task in the datasets containing transitive relations.

Motivation & Objective

• To address the gap in existing knowledge graph embedding models that fail to effectively model transitivity, a common yet underrepresented relation pattern.
• To theoretically demonstrate that transitive relations can be modeled using idempotent transformations, specifically projections.
• To design a unified model, Rot-Pro, capable of inferring multiple relation patterns—including transitivity—simultaneously.
• To empirically validate that Rot-Pro effectively learns transitivity and achieves state-of-the-art performance on link prediction benchmarks containing transitive relations.
• To investigate the trade-off between modeling transitivity and preserving other relation patterns like symmetry, especially under phase constraints.

Proposed method

• Rot-Pro integrates relational rotation in the complex plane with projection matrices to model transitivity, where projection matrices are constrained to be idempotent and similar to diagonal matrices with 0s and 1s on the diagonal.
• The model uses a score function based on rotated head entity representations projected via relation-specific matrices, combining rotational dynamics with projection-based idempotency.
• To ensure proper modeling of transitivity, the relational rotation phase is restricted to (−π/2, π/2) during training to avoid local optima at π, which can misrepresent transitive chains.
• The model’s expressiveness is theoretically proven to support symmetry, asymmetry, inversion, composition, and transitivity through the combination of rotation and projection.
• The embedding space is initialized with a range of (−π, π), but training with a restricted phase range improves convergence and correctness of transitive chain representations.
• The model is trained using negative sampling and a margin-based loss function, with ablation studies to evaluate the impact of phase constraints on different relation patterns.

Experimental results

Research questions

• RQ1 Can transitive relations in knowledge graphs be effectively modeled using projection-based idempotent transformations?
• RQ2 How can projection be combined with relational rotation to simultaneously model multiple relation patterns including transitivity?
• RQ3 What impact does constraining the relational rotation phase to (−π/2, π/2) have on the model’s ability to learn transitive chains and avoid local optima?
• RQ4 To what extent does Rot-Pro outperform existing models in link prediction tasks involving transitive relations?
• RQ5 What trade-offs exist between modeling transitivity and preserving other relation patterns such as symmetry when applying phase constraints?

Key findings

• Rot-Pro achieves state-of-the-art performance on the Countries dataset, which contains transitive relations, outperforming existing models in link prediction.
• The model successfully learns transitive chains, as evidenced by correct alignment of entity embeddings along a line in the rotated space when the phase is restricted to (−π/2, π/2).
• Without phase constraints, the model often converges to a local optimum at π, resulting in incorrect representations of transitive triples, such as (e1,r,e3) and (e2,r,e4) in a chain.
• The restriction of the relational rotation phase to (−π/2, π/2) significantly improves the model’s ability to represent transitive relations correctly, as shown in visualization and loss curve analysis.
• The model’s performance on symmetric relations degrades when phase constraints are applied, indicating a trade-off between modeling transitivity and preserving symmetry.
• Ablation studies confirm that while phase constraints enhance transitivity modeling, they negatively affect the learning of symmetric relations, suggesting a need for adaptive or attention-based mechanisms in future work.