[Paper Review] Sequence-to-Sequence Knowledge Graph Completion and Question Answering

Apoorv Saxena, Adrian Kochsiek+1 more|arXiv (Cornell University)|Mar 19, 2022

Advanced Graph Neural Networks11 citations

TL;DR

This paper proposes KGT5, a sequence-to-sequence Transformer model that treats knowledge graph completion (KGC) and question answering (KGQA) as text generation tasks. By training a T5-small-style model on link prediction via autoregressive decoding, KGT5 achieves state-of-the-art performance on KGC and KGQA with up to 98% smaller model size than conventional KGEs, while maintaining tractable inference and enabling unified, end-to-end training across tasks.

ABSTRACT

Knowledge graph embedding (KGE) models represent each entity and relation of a knowledge graph (KG) with low-dimensional embedding vectors. These methods have recently been applied to KG link prediction and question answering over incomplete KGs (KGQA). KGEs typically create an embedding for each entity in the graph, which results in large model sizes on real-world graphs with millions of entities. For downstream tasks these atomic entity representations often need to be integrated into a multi stage pipeline, limiting their utility. We show that an off-the-shelf encoder-decoder Transformer model can serve as a scalable and versatile KGE model obtaining state-of-the-art results for KG link prediction and incomplete KG question answering. We achieve this by posing KG link prediction as a sequence-to-sequence task and exchange the triple scoring approach taken by prior KGE methods with autoregressive decoding. Such a simple but powerful method reduces the model size up to 98% compared to conventional KGE models while keeping inference time tractable. After finetuning this model on the task of KGQA over incomplete KGs, our approach outperforms baselines on multiple large-scale datasets without extensive hyperparameter tuning.

Motivation & Objective

• To address the scalability, quality, versatility, and simplicity limitations of existing knowledge graph embedding (KGE) models.
• To unify knowledge graph completion and question answering into a single, scalable, and end-to-end trainable framework.
• To reduce model size and inference cost while maintaining or improving performance on large-scale knowledge graphs.
• To evaluate whether pretraining on link prediction is more effective than generic language modeling for knowledge-intensive tasks like KGQA.
• To investigate the trade-off between memorization and generalization in low-capacity models for incomplete KGs.

Proposed method

• The model uses an off-the-shelf encoder-decoder Transformer (T5-small architecture) to perform sequence-to-sequence prediction for knowledge graph triples.
• Link prediction is reformulated as a text generation task: given a relation and one entity (head or tail), the model autoregressively generates the missing entity.
• The model is first pretrained on a large-scale knowledge graph using a masked span prediction objective over triple sequences.
• For question answering, the same model is fine-tuned on QA pairs, with regularization from the link prediction objective to preserve relational knowledge.
• Entity representations are derived from their textual mentions, enabling compositional encoding without dedicated entity embeddings.
• Model size is drastically reduced by avoiding per-entity embedding vectors, relying instead on autoregressive decoding over token sequences.

Experimental results

Research questions

• RQ1 Can knowledge graph completion be effectively reformulated as a sequence-to-sequence generation task using a standard Transformer encoder-decoder architecture?
• RQ2 Does a unified seq2seq model trained on link prediction outperform specialized KGE models in terms of both performance and model efficiency?
• RQ3 Can such a model generalize well to incomplete knowledge graphs in question-answering tasks without extensive hyperparameter tuning?
• RQ4 How does pretraining on link prediction compare to generic language modeling for downstream KGQA tasks?
• RQ5 To what extent does model capacity limit memorization of known facts, especially in low-parameter models?

Key findings

• KGT5 achieves state-of-the-art performance on knowledge graph completion, with a test MRR of 0.300 on Wikidata5M, outperforming ComplEx in inference efficiency despite having only 60M parameters versus 614M.
• The model reduces model size by up to 98% compared to conventional KGEs while maintaining competitive performance on large-scale KGs with up to 90 million entities.
• On KGQA benchmarks, KGT5 achieves Hits@1 scores of 56.1 on WQSP and 36.5 on CWQ when fully pretrained on the KG, outperforming T5-small fine-tuned on QA alone.
• Pretraining on link prediction is significantly more effective than generic language modeling for KGQA, as shown by KGT5's superior performance compared to T5-small fine-tuned on QA.
• The model shows poor performance in the full-KG setting, with Hits@1 of 56.1 on WQSP, indicating limited memorization ability due to low model capacity (60M parameters).
• The train MRR of KGT5 (0.304) is significantly lower than ComplEx (0.721), confirming that the model struggles to memorize training facts, which limits its performance on closed-book QA.