[論文レビュー] KC-GenRe: A Knowledge-constrained Generative Re-ranking Method Based on Large Language Models for Knowledge Graph Completion

Yilin Wang, Minghao Hu+4 more|arXiv (Cornell University)|Mar 26, 2024

Advanced Graph Neural Networks被引用数 6

ひとこと要約

KC-GenRe は、知識制約付き生成型 LLM アプローチを用いて上位 K 個の KG 完成候補を再ランク付けし、ミスマッチ、誤順、および抜けを解消し、4つのデータセットで最先端の結果を達成します。

ABSTRACT

The goal of knowledge graph completion (KGC) is to predict missing facts among entities. Previous methods for KGC re-ranking are mostly built on non-generative language models to obtain the probability of each candidate. Recently, generative large language models (LLMs) have shown outstanding performance on several tasks such as information extraction and dialog systems. Leveraging them for KGC re-ranking is beneficial for leveraging the extensive pre-trained knowledge and powerful generative capabilities. However, it may encounter new problems when accomplishing the task, namely mismatch, misordering and omission. To this end, we introduce KC-GenRe, a knowledge-constrained generative re-ranking method based on LLMs for KGC. To overcome the mismatch issue, we formulate the KGC re-ranking task as a candidate identifier sorting generation problem implemented by generative LLMs. To tackle the misordering issue, we develop a knowledge-guided interactive training method that enhances the identification and ranking of candidates. To address the omission issue, we design a knowledge-augmented constrained inference method that enables contextual prompting and controlled generation, so as to obtain valid rankings. Experimental results show that KG-GenRe achieves state-of-the-art performance on four datasets, with gains of up to 6.7% and 7.7% in the MRR and Hits@1 metric compared to previous methods, and 9.0% and 11.1% compared to that without re-ranking. Extensive analysis demonstrates the effectiveness of components in KG-GenRe.

研究の動機と目的

• Motivation: Knowledge graphs are often incomplete and need effective re-ranking of candidate tail entities to improve completion quality.
• Objective: Develop a generative LLM-based re-ranking method that (i) avoids exact text matching (mismatch), (ii) learns proper candidate order (misordering), and (iii) ensures all candidates are considered (omission).
• Objective: Leverage first-stage KG embeddings to guide ranking and retrieve contextual knowledge to constrain generation for valid rankings.

提案手法

• Formulate re-ranking as outputting an ordered sequence of candidate option identifiers rather than candidate names to avoid mismatch.
• Introduce knowledge-guided interactive training with query-candidate and candidate-candidate interactions to learn candidate plausibility and relative order, using a ranking loss that aligns LLM outputs with first-stage scores.
• Propose knowledge-augmented constrained inference with query-related and candidate-supporting prompts, plus constrained option generation to ensure complete and valid rankings.
• Fine-tune LLaMA-7b via QLORA on the KC-GenRe framework, using top-K candidates from a first-stage KGE model and instruction templates for input/output formatting.

実験結果

リサーチクエスチョン

• RQ1 How can generative LLMs be effectively used to re-rank KGC candidates while mitigating mismatch between generated text and KG entities?
• RQ2 Can knowledge-guided training and knowledge-augmented inference improve the stability and quality of LLM-based KGC re-ranking?
• RQ3 Does constraining the LLM output to candidate option identifiers and leveraging contextual prompts improve ranking accuracy and validity?

主な発見

モデル	Wiki27K MRR	Wiki27K Hits@1	Wiki27K Hits@3	Wiki27K Hits@10	FB15K-237-N MRR	FB15K-237-N Hits@1	FB15K-237-N Hits@3	FB15K-237-N Hits@10
TransE †	0.155	0.032	0.228	0.378	0.255	0.152	0.301	0.459
TransC †	0.175	0.124	0.215	0.339	0.233	0.129	0.298	0.395
ConvE †	0.226	0.164	0.244	0.354	0.273	0.192	0.305	0.429
WWV †	0.198	0.157	0.237	0.365	0.269	0.137	0.287	0.443
TuckER Balazevic et al.	0.249	0.185	0.269	0.385	0.309	0.227	0.340	0.474
RotatE Sun et al.	0.216	0.123	0.256	0.394	0.279	0.177	0.320	0.481
KG-BERT Yao et al.	0.192	0.119	0.219	0.352	0.203	0.139	0.201	0.403
PKGC Lv et al.	0.285	0.230	0.305	0.409	0.332	0.261	0.346	0.487
KC-GenRe	0.317	0.274	0.330	0.408	0.399	0.338	0.427	0.505

• KC-GenRe achieves state-of-the-art re-ranking performance on four datasets, with up to 6.7% absolute gains in MRR and 7.7% in Hits@1 over previous methods on curated data.
• On open KG datasets, KC-GenRe outperforms baselines by 2.1–3.5% in MRR and 2.1–3.8% in Hits@1.
• Ablation shows query-candidate interaction and candidate-candidate interaction significantly boost performance, and the combination of query-related and candidate-supporting prompts yields the best results.
• Constrained option generation is crucial; removing it hurts performance, indicating the need to align decoding with the candidate set.