[Paper Review] KGE-CL: Contrastive Learning of Tensor Decomposition Based Knowledge Graph Embeddings

Luo Zhiping, Wentao Xu+4 more|arXiv (Cornell University)|Dec 9, 2021

Topic Modeling4 citations

TL;DR

This paper proposes KGE-CL, a contrastive learning framework for tensor decomposition-based knowledge graph embeddings that enhances semantic similarity between related entities and entity-relation couples across different triples. By constructing positive instances based on shared entities or relations and applying a weighted contrastive loss, KGE-CL achieves new state-of-the-art performance on WN18RR, FB15k-237, and YAGO3-10, with 51.2% MRR and 46.8% Hits@1 on WN18RR.

ABSTRACT

Learning the embeddings of knowledge graphs (KG) is vital in artificial intelligence, and can benefit various downstream applications, such as recommendation and question answering. In recent years, many research efforts have been proposed for knowledge graph embedding (KGE). However, most previous KGE methods ignore the semantic similarity between the related entities and entity-relation couples in different triples since they separately optimize each triple with the scoring function. To address this problem, we propose a simple yet efficient contrastive learning framework for tensor decomposition based (TDB) KGE, which can shorten the semantic distance of the related entities and entity-relation couples in different triples and thus improve the performance of KGE. We evaluate our proposed method on three standard KGE datasets: WN18RR, FB15k-237 and YAGO3-10. Our method can yield some new state-of-the-art results, achieving 51.2% MRR, 46.8% Hits@1 on the WN18RR dataset, 37.8% MRR, 28.6% Hits@1 on FB15k-237 dataset, and 59.1% MRR, 51.8% Hits@1 on the YAGO3-10 dataset.

Motivation & Objective

• Address the limitation in existing knowledge graph embedding (KGE) methods that fail to capture semantic similarity between related entities or entity-relation couples across different triples.
• Improve the expressiveness of tensor decomposition-based KGE models by explicitly modeling cross-triple semantic relationships.
• Develop a general, efficient contrastive learning framework compatible with existing tensor decomposition-based KGE methods.
• Enhance link prediction performance by shortening semantic distances between related entities and entity-relation couples in the embedding space.
• Enable sparse embedding learning to reduce storage usage while maintaining high performance, benefiting large-scale knowledge graphs.

Proposed method

• Construct positive instances for entities sharing the same entity-relation couple and for entity-relation couples sharing the same tail entity.
• Design a weighted contrastive loss function that dynamically controls the contribution of each positive instance to the overall loss.
• Integrate the contrastive learning framework into existing tensor decomposition-based KGE models (e.g., RESCAL, ComplEx) without architectural changes.
• Apply contrastive learning during training to minimize the distance between positive pairs (semantically similar entities/couples) and maximize distance between negative pairs.
• Utilize T-SNE visualization to analyze and validate the clustering of entity-relation couples in the embedding space.
• Conduct ablation studies and sparsity analysis to evaluate the impact of positive instances and embedding sparsity on model performance.

Experimental results

Research questions

• RQ1 Can modeling cross-triple semantic similarity between related entities and entity-relation couples improve knowledge graph embedding performance?
• RQ2 How effective is the proposed contrastive learning framework in capturing semantic relationships beyond individual triples?
• RQ3 Does the inclusion of positive instances based on shared entities or relations lead to consistent performance gains across different KGE models and datasets?
• RQ4 To what extent does the weighted contrastive loss improve generalization and robustness compared to standard contrastive learning?
• RQ5 Can the proposed framework maintain high performance under sparse embedding settings, crucial for real-world knowledge graphs?

Key findings

• KGE-CL achieves a new state-of-the-art performance of 51.2% MRR and 46.8% Hits@1 on the WN18RR dataset, surpassing previous methods.
• On FB15k-237, the method achieves 37.8% MRR and 28.6% Hits@1, setting a new SOTA result.
• On YAGO3-10, KGE-CL obtains 59.1% MRR and 51.8% Hits@1, demonstrating strong generalization across diverse knowledge graph structures.
• Ablation studies confirm that the inclusion of positive instances significantly improves performance, with consistent gains across multiple KGE variants.
• Visualization using T-SNE shows that KGE-CL better clusters entity-relation couples with the same tail entity compared to baseline models.
• Sparsity analysis reveals that KGE-CL embeddings maintain higher performance under sparse settings than DURA, indicating better storage efficiency for large-scale KGs.