[Paper Review] Lifelong Embedding Learning and Transfer for Growing Knowledge Graphs

Yuanning Cui, Yuxin Wang+5 more|arXiv (Cornell University)|Nov 29, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes LKGE, a lifelong knowledge graph embedding model that enables continual learning on growing KGs through a masked KG autoencoder, knowledge embedding transfer, and regularization to prevent catastrophic forgetting. The model achieves state-of-the-art performance in link prediction, forward knowledge transfer, and training efficiency across four diverse growth scenarios, outperforming existing baselines on entity-centric, relation-centric, fact-centric, and hybrid KG growth datasets.

ABSTRACT

Existing knowledge graph (KG) embedding models have primarily focused on static KGs. However, real-world KGs do not remain static, but rather evolve and grow in tandem with the development of KG applications. Consequently, new facts and previously unseen entities and relations continually emerge, necessitating an embedding model that can quickly learn and transfer new knowledge through growth. Motivated by this, we delve into an expanding field of KG embedding in this paper, i.e., lifelong KG embedding. We consider knowledge transfer and retention of the learning on growing snapshots of a KG without having to learn embeddings from scratch. The proposed model includes a masked KG autoencoder for embedding learning and update, with an embedding transfer strategy to inject the learned knowledge into the new entity and relation embeddings, and an embedding regularization method to avoid catastrophic forgetting. To investigate the impacts of different aspects of KG growth, we construct four datasets to evaluate the performance of lifelong KG embedding. Experimental results show that the proposed model outperforms the state-of-the-art inductive and lifelong embedding baselines.

Motivation & Objective

• To address the challenge of continual knowledge graph embedding in real-world, evolving KGs where new facts, entities, and relations emerge over time.
• To enable efficient knowledge transfer from previously learned embeddings to new, unseen entities and relations without retraining from scratch.
• To prevent catastrophic forgetting during incremental updates to old entity and relation embeddings.
• To evaluate model performance across diverse growth patterns—entity-centric, relation-centric, fact-centric, and hybrid—beyond idealized assumptions in prior datasets.
• To improve learning efficiency and maintain high predictive accuracy across multiple KG snapshots.

Proposed method

• A masked KG autoencoder is used to reconstruct new facts by masking and predicting entities or relations, enabling local knowledge transfer between old and new nodes.
• An embedding transfer strategy initializes new entity and relation embeddings using pre-trained knowledge from previous snapshots, accelerating convergence and stabilizing the embedding space.
• An embedding regularization method balances learning on new facts with retention of old knowledge, mitigating catastrophic forgetting during incremental updates.
• The model is trained incrementally on growing KG snapshots, updating embeddings only for affected entities while preserving the rest.
• The framework supports forward and backward knowledge transfer by maintaining consistent representations across time.
• Four new benchmark datasets are constructed to evaluate performance under different growth patterns: entity-centric, relation-centric, fact-centric, and hybrid.

Experimental results

Research questions

• RQ1 How can a KG embedding model effectively learn new facts while retaining previously acquired knowledge in a growing KG?
• RQ2 To what extent can knowledge transfer from existing embeddings improve the learning of new entities and relations in a lifelong learning setting?
• RQ3 How does the proposed regularization strategy mitigate catastrophic forgetting during incremental updates to old embeddings?
• RQ4 How does the model's performance vary across different types of KG growth—entity-centric, relation-centric, fact-centric, and hybrid?
• RQ5 What is the trade-off between learning efficiency and predictive accuracy in lifelong KG embedding compared to retraining or fine-tuning baselines?

Key findings

• LKGE achieves the highest mean reciprocal rank (MRR) on all four datasets, with MRR scores of 0.227 on Entity, 0.367 on Relation, 0.372 on Fact, and 0.367 on Hybrid, outperforming all baselines.
• The model demonstrates strong forward knowledge transfer (FWT), with FWT scores significantly higher than competitors, especially on Relation and Hybrid datasets where schema changes occur.
• Backward knowledge transfer (BWT) is effectively preserved, with LKGE achieving positive BWT scores across all datasets, indicating stable retention of old knowledge.
• Training time is substantially reduced compared to retraining and snapshot baselines, with LKGE showing the most significant efficiency gain on later snapshots due to embedding transfer.
• Ablation studies confirm that removing the masked autoencoder or regularization leads to significant performance drops, highlighting their critical role in knowledge retention and learning stability.
• The embedding transfer strategy improves performance on Relation and Hybrid datasets, where schema shifts occur, but is less critical on Entity, where high plasticity is needed for massive new entities.