[论文解读] Towards Foundation Models for Knowledge Graph Reasoning

Mikhail Galkin, Xinyu Yuan+3 more|arXiv (Cornell University)|Oct 6, 2023

Advanced Graph Neural Networks被引用 9

一句话总结

Ultra 学习通用、可迁移的 KG 表征，能够对具有任意实体和关系词汇表的未见图实现零样本泛化；先在少数图上进行预训练，再进行微调，相对于针对特定图的基线获得显著提升。

ABSTRACT

Foundation models in language and vision have the ability to run inference on any textual and visual inputs thanks to the transferable representations such as a vocabulary of tokens in language. Knowledge graphs (KGs) have different entity and relation vocabularies that generally do not overlap. The key challenge of designing foundation models on KGs is to learn such transferable representations that enable inference on any graph with arbitrary entity and relation vocabularies. In this work, we make a step towards such foundation models and present ULTRA, an approach for learning universal and transferable graph representations. ULTRA builds relational representations as a function conditioned on their interactions. Such a conditioning strategy allows a pre-trained ULTRA model to inductively generalize to any unseen KG with any relation vocabulary and to be fine-tuned on any graph. Conducting link prediction experiments on 57 different KGs, we find that the zero-shot inductive inference performance of a single pre-trained ULTRA model on unseen graphs of various sizes is often on par or better than strong baselines trained on specific graphs. Fine-tuning further boosts the performance.

研究动机与目标

• Motivation: enable transfer learning across knowledge graphs with different vocabularies.
• Goal: learn invariant relational structure representations that generalize to unseen entities and relations.
• Aim: build a foundation-model-style KG reasoning system that can be pre-trained and fine-tuned on downstream graphs.
• Demonstrate zero-shot transfer and fine-tuning benefits across a large suite of KG datasets.

提出的方法

• Construct a graph of relations from any KG to capture four fundamental interactions: tail-to-head, head-to-head, head-to-tail, tail-to-tail.
• Learn conditional relative relation representations using a labeling trick on the relation graph conditioned on the query relation.
• Use these conditional relation representations as inputs to an inductive link predictor on the original graph to perform KG completion.
• Pre-train Ultra on multiple KGs to capture transferable relational invariances; fine-tune on target graphs for improved performance.
• Parameterization avoids graph-specific entity/relation embeddings and input features, enabling zero-shot generalization.

实验结果

研究问题

• RQ1 Can a single pre-trained Ultra model inductively generalize to unseen KGs with arbitrary relation vocabularies (zero-shot)?
• RQ2 What gains are achieved by fine-tuning Ultra on target graphs after zero-shot transfer?
• RQ3 How does a pre-trained Ultra compare to models trained from scratch on each target graph?
• RQ4 Does increasing the mixture of pre-training graphs improve zero-shot transfer performance?

主要发现

• Zero-shot Ultra often matches or exceeds strong baselines trained on specific graphs, especially on smaller inductive graphs.
• Fine-tuning Ultra provides additional performance gains beyond zero-shot results.
• On average, Ultra outperforms supervised baselines in zero-shot scenarios by 15%.
• Fine-tuning yields about a 10% relative improvement on average across evaluated graphs.
• Ultra demonstrates transfer to 50+ KG benchmarks (1k–120k nodes; 5k–1M edges) with competitive results.