[Paper Review] Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata

Bohui Zhang, Ioannis Reklos+3 more|arXiv (Cornell University)|Sep 15, 2023

Topic Modeling4 citations

TL;DR

This paper proposes LLMKE, a pipeline that uses large language models (LLMs) like GPT-3.5-turbo and GPT-4 to predict Wikidata object entities for given subject-relation pairs, combining in-context learning with Wikidata entity mapping. It achieved a macro-averaged F1-score of 0.701 in the ISWC 2023 LM-KBC Challenge, demonstrating LLMs' potential for collaborative knowledge base completion despite significant performance variation across domains.

ABSTRACT

In this work, we explore the use of Large Language Models (LLMs) for knowledge engineering tasks in the context of the ISWC 2023 LM-KBC Challenge. For this task, given subject and relation pairs sourced from Wikidata, we utilize pre-trained LLMs to produce the relevant objects in string format and link them to their respective Wikidata QIDs. We developed a pipeline using LLMs for Knowledge Engineering (LLMKE), combining knowledge probing and Wikidata entity mapping. The method achieved a macro-averaged F1-score of 0.701 across the properties, with the scores varying from 1.00 to 0.328. These results demonstrate that the knowledge of LLMs varies significantly depending on the domain and that further experimentation is required to determine the circumstances under which LLMs can be used for automatic Knowledge Base (e.g., Wikidata) completion and correction. The investigation of the results also suggests the promising contribution of LLMs in collaborative knowledge engineering. LLMKE won Track 2 of the challenge. The implementation is available at https://github.com/bohuizhang/LLMKE.

Motivation & Objective

• To investigate the feasibility of using large language models (LLMs) for automated knowledge base completion in Wikidata.
• To evaluate how LLMs perform across diverse Wikidata relations with varying knowledge coverage and domain specificity.
• To develop a pipeline that combines in-context learning with Wikidata entity disambiguation for accurate object prediction.
• To assess the role of LLMs in improving Wikidata quality by identifying incomplete or outdated triples.
• To explore the potential of LLMs as collaborative tools in knowledge engineering, especially in detecting and suggesting corrections for knowledge gaps.

Proposed method

• The method employs in-context learning with few-shot prompting to elicit object predictions from LLMs given subject and relation pairs.
• Predicted object strings are mapped to Wikidata QIDs using the wbsearchentities API, which relies on labels and aliases for disambiguation.
• A two-stage pipeline is used: first, LLMs generate candidate object strings; second, these are matched to Wikidata entities via label and alias-based search.
• The approach leverages both GPT-3.5-turbo and GPT-4 to compare performance across models and identify complementary knowledge strengths.
• The system evaluates predictions against ground truth Wikidata triples, computing F1 scores per relation and macro-averaged across all properties.
• Context from Wikipedia is used in some experiments to improve LLM reasoning, though this occasionally degrades performance due to knowledge mismatches.

Experimental results

Research questions

• RQ1 Can LLMs reliably predict Wikidata object entities for given subject-relation pairs using few-shot in-context learning?
• RQ2 How does LLM performance vary across different Wikidata relations, and what factors influence this variation?
• RQ3 To what extent do knowledge gaps between Wikipedia, Wikidata, and LLMs affect prediction accuracy?
• RQ4 Can LLMs be used to detect and suggest corrections for incomplete or outdated Wikidata triples?
• RQ5 Can ensemble use of multiple LLMs improve overall performance by leveraging complementary knowledge?

Key findings

• The LLMKE pipeline achieved a macro-averaged F1-score of 0.701 across all Wikidata properties, with scores ranging from 0.328 to 1.00.
• GPT-4 achieved a perfect F1-score of 1.00 on the PersonHasNobelPrize relation, indicating strong performance on well-known, high-precision facts.
• The lowest-performing relation, PersonHasEmployer, achieved an F1-score of only 0.328, highlighting LLMs’ limitations in less common or domain-specific knowledge.
• The wbsearchentities API often failed to match LLM-predicted strings due to missing labels or aliases, limiting disambiguation accuracy.
• A significant knowledge gap was observed between Wikipedia and Wikidata, where LLMs correctly predicted updated facts (e.g., Exor as parent of Ferrari) that Wikidata had not yet reflected.
• Surprisingly, GPT-3.5-turbo outperformed GPT-4 on two relations, suggesting that different LLMs hold distinct knowledge, supporting ensemble approaches.