[Paper Review] Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering

Lars‐Peter Meyer, Johannes Frey+5 more|arXiv (Cornell University)|Aug 31, 2023

Topic Modeling4 citations

TL;DR

This paper introduces LLM-KG-Bench, a modular, scalable benchmark framework for evaluating Large Language Models (LLMs) in knowledge graph engineering (KGE) tasks. It supports automated evaluation of LLMs across three core challenges—syntax correction, fact extraction, and synthetic dataset generation—demonstrating that while LLMs show promise, they remain inconsistent in zero-shot KGE, especially with syntax and structural accuracy.

ABSTRACT

As the field of Large Language Models (LLMs) evolves at an accelerated pace, the critical need to assess and monitor their performance emerges. We introduce a benchmarking framework focused on knowledge graph engineering (KGE) accompanied by three challenges addressing syntax and error correction, facts extraction and dataset generation. We show that while being a useful tool, LLMs are yet unfit to assist in knowledge graph generation with zero-shot prompting. Consequently, our LLM-KG-Bench framework provides automatic evaluation and storage of LLM responses as well as statistical data and visualization tools to support tracking of prompt engineering and model performance.

Motivation & Objective

• Address the lack of standardized, automated evaluation frameworks for LLMs in knowledge graph engineering (KGE) tasks.
• Enable continuous, repeatable benchmarking of LLMs across diverse KGE workloads with configurable task sizing.
• Support tracking of model performance and prompt engineering outcomes through persistent result storage and visualization.
• Identify limitations of current LLMs in zero-shot KGE, particularly in syntactic correctness and structural fidelity.
• Establish a foundation for community-driven expansion of KGE-specific benchmarks and model comparisons.

Proposed method

• Design a modular framework with dedicated benchmark tasks and LLM model connectors abstracting model interactions.
• Implement a standardized `generate_text` interface for LLM connectors to enable consistent prompt delivery and response collection.
• Define task-specific evaluation functions that compute scores such as F1 for triple matching and normalized error for object count deviation.
• Support configurable problem sizes (e.g., number of entities) to study context length effects on LLM performance.
• Integrate result persistence and visualization using seaborn for statistical analysis and trend tracking.
• Enable dialog-based evaluation by designing tasks that allow LLMs to iteratively correct responses based on feedback.

Figure 1 : Basic LLM-KG-Bench framework architecture. The Benchmark runner takes a benchmark configuration and organizes the repeated execution of benchmark tasks with LLM model connectors and given size parameters. Results generated get stored and can be visualized.

Experimental results

Research questions

• RQ1 How do leading LLMs perform in correcting syntactic errors in Turtle-formatted knowledge graphs?
• RQ2 To what extent can LLMs accurately extract structured facts from unstructured, PDF-extracted plaintext into valid RDF/Turtle?
• RQ3 How reliably can LLMs generate synthetic knowledge graphs with specified numbers of entities and relationships?
• RQ4 How does the size of the input or target knowledge graph affect LLM performance across KGE tasks?
• RQ5 Can automated benchmarking frameworks like LLM-KG-Bench detect and quantify model inconsistencies in zero-shot KGE scenarios?

Key findings

• GPT-3.5 frequently returned empty responses when asked to fix Turtle syntax, resulting in zero F1 scores due to incorrect claims of correctness.
• Claude-1.3 and GPT-4 outperformed GPT-3.5 in Turtle error correction, with GPT-4 showing higher mean F1 but more unparseable outputs.
• GPT models achieved better F1 scores in fact extraction from plaintext than Claude-1.3, though GPT-4’s performance was skewed by a single high-quality response.
• GPT-3.5 showed more consistent output quality in fact extraction, with no unparseable responses, leading to a better median F1 despite lower mean.
• In synthetic dataset generation, the relative error in person count increased with target size, with GPT-4 showing higher variance and more deviations from target counts.
• The framework successfully captured performance degradation with increasing problem size, particularly in object count accuracy, highlighting context length limitations.

(a) Turtle Fixing